00010000:00010000:2.0:1788572475.673783:0:108846:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70204a00/0x58a604dcef3520ff lrc: 0/0,0 mode: --/PW res: [0x20000041c:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221791e6a expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572475.673857:0:108846:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6733b480 x1875449863805056/t4294983638(4294983638) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572475.673869:0:108846:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572475.673874:0:108846:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863805824/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572475.673895:0:108846:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108846:1875449863805824:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572475.673960:0:108846:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70555a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.676529:0:108846:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108846:1875449863805824:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572475.676565:0:108846:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6733b480 x1875449863805056/t4294983638(4294983638) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572475.676629:0:108846:0:(file.c:479:ll_file_release()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), flags 154000 00000002:00100000:2.0:1788572475.676682:0:108846:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617df80 x1875449863804672/t4294983637(4294983637) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572475.676693:0:108846:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572475.676699:0:108846:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863806080/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572475.676718:0:108846:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108846:1875449863806080:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572475.676811:0:108846:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70555a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.686295:0:108846:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108846:1875449863806080:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572475.686333:0:108846:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617df80 x1875449863804672/t4294983637(4294983637) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572475.716987:0:108818:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572475.804205:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2395:1875449863806720:192.168.202.132@tcp:3:lfs.0 00000080:00200000:2.0:1788572475.829133:0:108851:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:2.0:1788572475.829152:0:108851:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041c:0x2:0x0] ignore/verify layout 0/0, layout version 5 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572475.829157:0:108851:0:(file.c:2238:ll_file_io_generic()) f59a.sanity-sec: 1 io complete with rc: 0, result: 5000, restart: 0 00000020:00200000:2.0:1788572475.829161:0:108851:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572475.829171:0:108851:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 5000 00000080:00200080:2.0:1788572475.829189:0:108851:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=400866b1 arg=0 00000080:00200080:2.0:1788572475.842044:0:108851:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=c00c66f3 arg=7ffe9916fbc4 00010000:00010000:2.0:1788572475.842070:0:108851:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc2c00/0x58a604dcef352114 lrc: 2/0,0 mode: PW/PW res: [0x20000041c:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a221791ea9 expref: -99 pid: 108851 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.842125:0:108851:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc2c00/0x58a604dcef352114 lrc: 1/0,0 mode: --/PW res: [0x20000041c:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221791ea9 expref: -99 pid: 108851 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.842132:0:108851:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572475.842156:0:108851:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108851:1875449863806976:192.168.202.132@tcp:103:lfs.0 00000100:00100000:2.0:1788572475.842289:0:108851:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70555840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.852262:0:108851:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108851:1875449863806976:192.168.202.132@tcp:103:lfs.0 00010000:00010000:2.0:1788572475.852296:0:108851:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc2c00/0x58a604dcef352114 lrc: 0/0,0 mode: --/PW res: [0x20000041c:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221791ea9 expref: -99 pid: 108851 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572475.852397:0:108851:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67048a80 x1875449863806592/t4294983642(4294983642) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572475.852409:0:108851:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572475.852415:0:108851:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863807232/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572475.852437:0:108851:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108851:1875449863807232:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572475.852511:0:108851:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70555840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.858222:0:108851:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:108851:1875449863807232:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572475.858262:0:108851:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67048a80 x1875449863806592/t4294983642(4294983642) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572475.858315:0:108851:0:(file.c:479:ll_file_release()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), flags 154000 00010000:00010000:2.0:1788572475.858345:0:108851:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc3e00/0x58a604dcef35210d lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e94 expref: -99 pid: 108851 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788572475.882603:0:108853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) for lock 0x58a604dcef3520f1 00000080:00010000:1.0:1788572475.882610:0:108853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520f1 from it ffff9e03832ffcd8 00010000:00010000:1.0:1788572475.882614:0:108853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572475.882623:0:108853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000080:00200080:2.0:1788572475.948287:0:108853:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=400866b1 arg=1 00000080:00200008:2.0:1788572475.948314:0:108853:0:(file.c:6296:ll_getattr_dentry()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572475.948324:0:108853:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08),name=f59a.sanity-sec 00000002:00010000:2.0:1788572475.948465:0:108853:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x2:0x0]) in obj [0x20000041c:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572475.948488:0:108853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572475.948504:0:108853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572475.948513:0:108853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041c:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572475.948521:0:108853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572475.948525:0:108853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) for lock 0x58a604dcef3520f1 00000080:00010000:2.0:1788572475.948533:0:108853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520f1 from it ffff9e03832ffcd8 00010000:00010000:2.0:1788572475.948537:0:108853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572475.948554:0:108853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 12 lru_type: 1 00000080:00200008:2.0:1788572475.948595:0:108853:0:(file.c:2483:do_file_read_iter()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0], ppos: 0, count: 67108864 00000080:00200000:2.0:1788572475.948609:0:108853:0:(file.c:2090:ll_file_io_generic()) f59a.sanity-sec: read ppos: 0, bytes: 67108864 00000080:00200000:2.0:1788572475.948622:0:108853:0:(file.c:1848:ll_io_set_mirror()) f59a.sanity-sec: desiginated mirror: 1 00000020:00200000:2.0:1788572475.948630:0:108853:0:(cl_io.c:214:cl_io_rw_init()) header@0000000074c825bf[0x110, 2, [0x20000041c:0x2:0x0] hash] 00000020:00200000:2.0:1788572475.948632:0:108853:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 67108864) 0 0 00000080:00200000:2.0:1788572475.948703:0:108853:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041c:0x2:0x0] ignore/verify layout 0/0, layout version 5 restore needed 0 00000080:00200000:2.0:1788572475.948884:0:108853:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:2.0:1788572475.948892:0:108853:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:2.0:1788572475.948897:0:108853:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572475.948915:0:108853:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:2.0:1788572475.948930:0:108853:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1788572475.948935:0:108853:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc482ed8b0, osc lock ffff92fc58370130, flags 0x0 00000080:00200000:2.0:1788572475.948944:0:108853:0:(vvp_io.c:835:vvp_io_read_start()) f59a.sanity-sec: read [0, 67108864) 00000001:00010000:2.0:1788572475.948953:0:108853:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041c:0x2:0x0] 00000008:00010000:2.0:1788572475.948959:0:108853:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572475.948961:0:108853:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc482edb30, osc lock ffff92fc58371a20, flags 0x400001002 00010000:00010000:2.0:1788572475.948976:0:108853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.948987:0:108853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572475.948999:0:108853:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572475.949002:0:108853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572475.949460:0:108853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572475.949481:0:108853:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572475.949489:0:108853:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041c:0x2:0x0] updating i_size 5000 i_blocks 16 00000020:00010000:2.0:1788572475.949494:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc67175030 (ffff92fc6713bc68/0) at cl_lock_release():245 00000020:00010000:2.0:1788572475.949498:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67175030 (ffff92fc6713bc68/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572475.949503:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc482edb30 (ffff92fc6713bd88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572475.949507:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67175030 (ffff92fc6713bc68/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572475.949510:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc482edb30 (ffff92fc6713bc68/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572475.949526:0:108853:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:2.0:1788572475.949532:0:108853:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08), size=67108864 (max 536870912), offset=0=0x0, pages 16384 (max 131072), lockless, parallel 00000020:00200000:2.0:1788572475.949595:0:108853:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000080:00200000:2.0:1788572475.949598:0:108853:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 5000 count 5000 file_offset 5000 00000020:00010000:2.0:1788572475.949605:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc7002da88 (ffff92fc6713bc68/0) at cl_lock_release():245 00000020:00010000:2.0:1788572475.949608:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7002da88 (ffff92fc6713bc68/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572475.949610:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc482ed8b0 (ffff92fc6713bd88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572475.949613:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7002da88 (ffff92fc6713bc68/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572475.949616:0:108853:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc482ed8b0 (ffff92fc6713bc68/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572475.949621:0:108853:0:(vvp_io.c:533:vvp_io_advance()) advancing 5000 bytes 00000020:00200000:2.0:1788572475.949626:0:108853:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572475.949867:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc508a5600 00000008:00100000:2.0:1788572475.949912:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617d180 - object 0x280000401:550 offset 0<>8192 00000008:00010000:2.0:1788572475.958171:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc508a5600 00000020:00200000:2.0:1788572475.958184:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:2.0:1788572475.958187:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1788572475.958197:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572475.958219:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc508a5600 00000100:00100000:2.0:1788572475.958249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2396:1875449863807360:192.168.202.132@tcp:3:lfs.0 00000080:00200008:2.0:1788572475.978565:0:108853:0:(file.c:479:ll_file_release()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), flags 140000 00010000:00010000:2.0:1788572475.978614:0:108853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc3e00/0x58a604dcef35210d lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e94 expref: -99 pid: 108851 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200080:2.0:1788572476.112237:0:108855:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=400866b1 arg=2 00000080:00200008:2.0:1788572476.112261:0:108855:0:(file.c:6296:ll_getattr_dentry()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572476.112271:0:108855:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08),name=f59a.sanity-sec 00000002:00010000:2.0:1788572476.112321:0:108855:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x2:0x0]) in obj [0x20000041c:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572476.112342:0:108855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572476.112372:0:108855:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572476.112382:0:108855:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041c:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572476.112389:0:108855:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572476.112392:0:108855:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) for lock 0x58a604dcef3520f1 00000080:00010000:2.0:1788572476.112400:0:108855:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520f1 from it ffff9e038330fcd8 00010000:00010000:2.0:1788572476.112403:0:108855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572476.112412:0:108855:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00000080:00200008:2.0:1788572476.112445:0:108855:0:(file.c:2483:do_file_read_iter()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0], ppos: 0, count: 67108864 00000080:00200000:2.0:1788572476.112457:0:108855:0:(file.c:2090:ll_file_io_generic()) f59a.sanity-sec: read ppos: 0, bytes: 67108864 00000080:00200000:2.0:1788572476.112467:0:108855:0:(file.c:1848:ll_io_set_mirror()) f59a.sanity-sec: desiginated mirror: 2 00000020:00200000:2.0:1788572476.112475:0:108855:0:(cl_io.c:214:cl_io_rw_init()) header@0000000074c825bf[0x110, 2, [0x20000041c:0x2:0x0] hash] 00000020:00200000:2.0:1788572476.112477:0:108855:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 67108864) 0 0 00000080:00200000:2.0:1788572476.112483:0:108855:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041c:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572476.112497:0:108855:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:2.0:1788572476.112505:0:108855:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:2.0:1788572476.112508:0:108855:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572476.112549:0:108855:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:2.0:1788572476.112562:0:108855:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1788572476.112567:0:108855:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc482ed8b0, osc lock ffff92fc58370130, flags 0x0 00000080:00200000:2.0:1788572476.112575:0:108855:0:(vvp_io.c:835:vvp_io_read_start()) f59a.sanity-sec: read [0, 67108864) 00000001:00010000:2.0:1788572476.112586:0:108855:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041c:0x2:0x0] 00000008:00010000:2.0:1788572476.112592:0:108855:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572476.112593:0:108855:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc482edb30, osc lock ffff92fc58371a20, flags 0x400001002 00010000:00010000:2.0:1788572476.112644:0:108855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.112655:0:108855:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572476.112711:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788572476.112770:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572476.112791:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2396:1875449863807488:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572476.116366:0:2396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.116476:0:2396:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.116497:0:2396:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.116515:0:2396:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572476.116530:0:2396:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 5000, setting rss=5000; kms=5000, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572476.116543:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572476.116565:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.116575:0:2396:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788572476.116760:0:108855:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041c:0x2:0x0] updating i_size 5000 i_blocks 32 00000020:00010000:1.0:1788572476.116767:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704762b0 (ffff92fc6721ad88/0) at cl_lock_release():245 00000020:00010000:1.0:1788572476.116771:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704762b0 (ffff92fc6721ad88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572476.116777:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc482edb30 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572476.116781:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704762b0 (ffff92fc6721ad88/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572476.116784:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc482edb30 (ffff92fc6721ad88/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572476.116799:0:108855:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:1.0:1788572476.116806:0:108855:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08), size=67108864 (max 536870912), offset=0=0x0, pages 16384 (max 131072), lockless, parallel 00000020:00200000:1.0:1788572476.116875:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000080:00200000:1.0:1788572476.116879:0:108855:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 5000 count 5000 file_offset 5000 00000020:00010000:1.0:1788572476.116886:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e35d8 (ffff92fc6721ad88/0) at cl_lock_release():245 00000020:00010000:1.0:1788572476.116888:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e35d8 (ffff92fc6721ad88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572476.116891:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc482ed8b0 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572476.116894:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e35d8 (ffff92fc6721ad88/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572476.116896:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc482ed8b0 (ffff92fc6721ad88/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572476.116903:0:108855:0:(vvp_io.c:533:vvp_io_advance()) advancing 5000 bytes 00000020:00200000:1.0:1788572476.116909:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1788572476.116929:0:103957:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc48227300 00000008:00100000:1.0:1788572476.116979:0:103957:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc6728bb80 - object 0x2c0000401:536 offset 0<>8192 00010000:00010000:2.0:1788572476.117249:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.117261:0:2396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572476.117285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2396:1875449863807488:192.168.202.132@tcp:101:lfs.0 00000008:00010000:1.0:1788572476.128672:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc48227300 00000020:00200000:1.0:1788572476.128687:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:1.0:1788572476.128693:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1788572476.128704:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788572476.128749:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc48227300 00000100:00100000:1.0:1788572476.128782:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2393:1875449863807616:192.168.202.132@tcp:3:lfs.0 00000080:00200000:2.0:1788572476.128837:0:108855:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:2.0:1788572476.128861:0:108855:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041c:0x2:0x0] ignore/verify layout 0/0, layout version 5 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572476.128866:0:108855:0:(file.c:2238:ll_file_io_generic()) f59a.sanity-sec: 1 io complete with rc: 0, result: 5000, restart: 0 00000020:00200000:2.0:1788572476.128870:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572476.128879:0:108855:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 5000 00000080:00200080:2.0:1788572476.128896:0:108855:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=400866b1 arg=0 00000080:00200080:2.0:1788572476.128988:0:108855:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=400866b1 arg=2 00000080:00200008:2.0:1788572476.128998:0:108855:0:(file.c:6296:ll_getattr_dentry()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572476.129005:0:108855:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08),name=f59a.sanity-sec 00000002:00010000:2.0:1788572476.129062:0:108855:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x2:0x0]) in obj [0x20000041c:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572476.129083:0:108855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572476.129107:0:108855:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572476.129116:0:108855:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041c:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572476.129123:0:108855:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572476.129127:0:108855:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x2:0x0](ffff92fc665e2a08) for lock 0x58a604dcef3520f1 00000080:00010000:2.0:1788572476.129135:0:108855:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520f1 from it ffff9e038330fcd8 00010000:00010000:2.0:1788572476.129139:0:108855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572476.129148:0:108855:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00000080:00200008:2.0:1788572476.129175:0:108855:0:(file.c:2483:do_file_read_iter()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0], ppos: 5000, count: 67108864 00000080:00200000:2.0:1788572476.129184:0:108855:0:(file.c:2090:ll_file_io_generic()) f59a.sanity-sec: read ppos: 5000, bytes: 67108864 00000080:00200000:2.0:1788572476.129191:0:108855:0:(file.c:1848:ll_io_set_mirror()) f59a.sanity-sec: desiginated mirror: 2 00000020:00200000:2.0:1788572476.129199:0:108855:0:(cl_io.c:214:cl_io_rw_init()) header@0000000074c825bf[0x110, 2, [0x20000041c:0x2:0x0] hash] 00000020:00200000:2.0:1788572476.129201:0:108855:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [5000, 67113864) 0 0 00000080:00200000:2.0:1788572476.129206:0:108855:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041c:0x2:0x0] ignore/verify layout 0/0, layout version 5 restore needed 0 00000080:00200000:2.0:1788572476.129220:0:108855:0:(file.c:2168:ll_file_io_generic()) Range lock [1, 16385] 00020000:00200000:2.0:1788572476.129227:0:108855:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 5000 chunk: [5000, 67113864) 67113864, 67108864 00020000:00200000:2.0:1788572476.129231:0:108855:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572476.129246:0:108855:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [1, 16385] 00000008:00010000:2.0:1788572476.129260:0:108855:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1788572476.129265:0:108855:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc482ed6b0, osc lock ffff92fc58371300, flags 0x0 00000080:00200000:2.0:1788572476.129271:0:108855:0:(vvp_io.c:835:vvp_io_read_start()) f59a.sanity-sec: read [5000, 67113864) 00000001:00010000:2.0:1788572476.129281:0:108855:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041c:0x2:0x0] 00000008:00010000:2.0:1788572476.129286:0:108855:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572476.129288:0:108855:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc482ed3b0, osc lock ffff92fc58370000, flags 0x400001002 00010000:00010000:2.0:1788572476.129303:0:108855:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572476.129315:0:108855:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572476.129326:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572476.129330:0:108855:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572476.129339:0:108855:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572476.129357:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572476.129363:0:108855:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041c:0x2:0x0] updating i_size 5000 i_blocks 32 00000020:00010000:2.0:1788572476.129367:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704762b0 (ffff92fc6721ad88/0) at cl_lock_release():245 00000020:00010000:2.0:1788572476.129371:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704762b0 (ffff92fc6721ad88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572476.129373:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc482ed3b0 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572476.129377:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704762b0 (ffff92fc6721ad88/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572476.129380:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc482ed3b0 (ffff92fc6721ad88/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572476.129457:0:108855:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:5000 00000080:00200000:2.0:1788572476.129462:0:108855:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08), size=67108864 (max 536870912), offset=5000=0x1388, pages 16384 (max 131072), lockless, parallel, unaligned 00000020:00010000:2.0:1788572476.129475:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e35d8 (ffff92fc6721ad88/0) at cl_lock_release():245 00000020:00010000:2.0:1788572476.129477:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e35d8 (ffff92fc6721ad88/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572476.129480:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc482ed6b0 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572476.129482:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e35d8 (ffff92fc6721ad88/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572476.129485:0:108855:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc482ed6b0 (ffff92fc6721ad88/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572476.129490:0:108855:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:2.0:1788572476.129496:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572476.129498:0:108855:0:(file.c:2205:ll_file_io_generic()) Range unlock [1, 16385] 00000080:00200000:2.0:1788572476.129502:0:108855:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041c:0x2:0x0] ignore/verify layout 0/0, layout version 5 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572476.129506:0:108855:0:(file.c:2238:ll_file_io_generic()) f59a.sanity-sec: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:2.0:1788572476.129508:0:108855:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572476.129511:0:108855:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200080:2.0:1788572476.129516:0:108855:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) cmd=400866b1 arg=0 00000080:00200008:2.0:1788572476.152345:0:108855:0:(file.c:479:ll_file_release()) START file f59a.sanity-sec:[0x20000041c:0x2:0x0](ffff92fc665e2a08), flags 140000 00010000:00010000:2.0:1788572476.152414:0:108855:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc3e00/0x58a604dcef35210d lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221791e94 expref: -99 pid: 108851 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000100:00100000:1.0:1788572480.011736:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2394:1875449863807872:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572480.011756:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2394:1875449863808128:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572480.011767:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c35a6bf6-0615-429a-bace-f01c5ef29804:2394:1875449863807744:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572480.011779:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2394:1875449863808000:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572480.011789:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2394:1875449863808256:192.168.202.132@tcp:400:kworker.0 00010000:00010000:2.0:1788572480.837895:0:108819:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572482.811625:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08),name=/ 00000002:00010000:2.0:1788572482.811661:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.811681:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572482.811904:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572482.811915:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.811923:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572482.811927:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) for lock 0x58a604dcef3520e3 00000080:00010000:2.0:1788572482.811935:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520e3 from it ffff9e0381d33b58 00010000:00010000:2.0:1788572482.811939:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572482.811947:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572482.811964:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572482.811971:0:109280:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59a.sanity-sec, flags=8 00000080:00200008:2.0:1788572482.811979:0:109280:0:(file.c:6296:ll_getattr_dentry()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572482.811984:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8),name=d59a.sanity-sec 00000002:00010000:2.0:1788572482.811996:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.812004:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572482.812013:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572482.812020:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.812024:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) 00000080:00010000:2.0:1788572482.812027:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) for lock 0x58a604dcef3520ea 00000080:00010000:2.0:1788572482.812031:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520ea from it ffff9e0381d33cb8 00010000:00010000:2.0:1788572482.812034:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572482.812042:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572482.812071:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08),name=/ 00000002:00010000:2.0:1788572482.812078:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.812085:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572482.812093:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572482.812099:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.812103:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572482.812105:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) for lock 0x58a604dcef3520e3 00000080:00010000:2.0:1788572482.812109:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520e3 from it ffff9e0381d33bf8 00010000:00010000:2.0:1788572482.812112:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572482.812119:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572482.812126:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572482.812130:0:109280:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59a.sanity-sec, flags=266 00000080:00200000:2.0:1788572482.812134:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572482.812140:0:109280:0:(file.c:1090:ll_file_open()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), flags 704400 00000002:00010000:2.0:1788572482.812154:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788572482.812200:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.812207:0:109280:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572482.812219:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572482.812231:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863808512/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572482.812240:0:109280:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572482.812260:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863808512:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572482.812502:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d09c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.815646:0:109280:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4871c800: committing for last_committed 4294983643 gen 1 00000100:00100000:2.0:1788572482.815667:0:109280:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc67049c00 x1875449863806336/t4294983641(4294983641) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572482.815678:0:109280:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff92fc4617d180 x1875449863808512/t4294983644(4294983644) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 3 fl Rpc:RPQU/604/0 rc 0/301 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572482.815691:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863808512:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572482.815882:0:109280:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815896:0:109280:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815906:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815913:0:109280:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815921:0:109280:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815927:0:109280:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366000 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572482.815933:0:109280:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815940:0:109280:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815947:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815953:0:109280:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572482.815959:0:109280:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.815966:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352122 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572482.815982:0:109280:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617d180 x1875449863808512/t4294983644(4294983644) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572482.815994:0:109280:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863808512/t4294983644(4294983644) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572482.816006:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572482.816027:0:109280:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041c:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572482.816048:0:109280:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863808512/t4294983644(4294983644) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572482.816115:0:109280:0:(file.c:6296:ll_getattr_dentry()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572482.816122:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8),name=d59a.sanity-sec 00000002:00010000:2.0:1788572482.816143:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.816157:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572482.816168:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572482.816175:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.816181:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) 00000080:00010000:2.0:1788572482.816184:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) for lock 0x58a604dcef3520ea 00000080:00010000:2.0:1788572482.816189:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520ea from it ffff9e0381d33cd8 00010000:00010000:2.0:1788572482.816192:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572482.816200:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572482.816281:0:109280:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788572482.816298:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x1:0x0]) in obj [0x20000041c:0x1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572482.816305:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572482.816313:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1788572482.817114:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863808640:192.168.202.132@tcp:37:rm.0 00000100:00100000:2.0:1788572482.817261:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d09c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.823815:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863808640:192.168.202.132@tcp:37:rm.0 00010000:00010000:2.0:1788572482.824545:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572482.824561:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:2.0:1788572482.824956:0:109280:0:(file.c:479:ll_file_release()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), flags 704000 00000002:00100000:2.0:1788572482.825024:0:109280:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617d180 x1875449863808512/t4294983644(4294983644) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572482.825036:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572482.825041:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863808896/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572482.825061:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863808896:192.168.202.132@tcp:35:rm.0 00000100:00100000:2.0:1788572482.825276:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d09c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.828822:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863808896:192.168.202.132@tcp:35:rm.0 00000002:00100000:2.0:1788572482.828858:0:109280:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617d180 x1875449863808512/t4294983644(4294983644) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572482.829039:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08),name=/ 00000002:00010000:2.0:1788572482.829059:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.829077:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572482.829090:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572482.829098:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.829109:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572482.829112:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) for lock 0x58a604dcef3520e3 00000080:00010000:2.0:1788572482.829120:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520e3 from it ffff9e0381d33bf8 00010000:00010000:2.0:1788572482.829194:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572482.829206:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572482.829221:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572482.829228:0:109280:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59a.sanity-sec, flags=266 00000080:00200000:2.0:1788572482.829235:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572482.829243:0:109280:0:(file.c:1090:ll_file_open()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), flags 704400 00000002:00010000:2.0:1788572482.829260:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1788572482.829305:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.829313:0:109280:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572482.829324:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572482.829332:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863809152/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572482.829424:0:109280:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572482.829446:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809152:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572482.830086:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d09c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.835866:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809152:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572482.835921:0:109280:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 4/1,0 mode: --/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.835934:0:109280:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 4/1,0 mode: --/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.835950:0:109280:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 4/1,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.835961:0:109280:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 4/1,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572482.835980:0:109280:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc4617df80 x1875449863809152/t4294983646(4294983646) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572482.835994:0:109280:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617df80 x1875449863809152/t4294983646(4294983646) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572482.836279:0:109280:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 3/1,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.836291:0:109280:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5deabc00 type 13 mode 16 res 8589935644/1 (0 0) 00000080:00200000:2.0:1788572482.836298:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572482.836322:0:109280:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041c:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572482.836328:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) for lock 0x58a604dcef352129 00000080:00010000:2.0:1788572482.836335:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352129 from it ffff9e0381d33bd0 00010000:00010000:2.0:1788572482.836419:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 3/1,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.836429:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572482.836461:0:109280:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617df80 x1875449863809152/t4294983646(4294983646) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572482.836666:0:109280:0:(file.c:6296:ll_getattr_dentry()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572482.836674:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8),name=d59a.sanity-sec 00000002:00010000:2.0:1788572482.836872:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041c:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.836944:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572482.836957:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572482.836964:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.836970:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) 00000080:00010000:2.0:1788572482.836973:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) for lock 0x58a604dcef3520ea 00000080:00010000:2.0:1788572482.836977:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520ea from it ffff9e0381d33cd8 00010000:00010000:2.0:1788572482.836980:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572482.836986:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572482.837234:0:109280:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff92fc4871c800) d59a.sanity-sec proj=0 00000080:00000004:2.0:1788572482.837239:0:109280:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc4871c800: age 4230, max_age 4237 00800000:00000004:2.0:1788572482.837251:0:109280:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc4871c800: age 4230, max_age 4237 00000100:00100000:2.0:1788572482.837289:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809280:192.168.202.132@tcp:41:rm.0 00000100:00100000:2.0:1788572482.840253:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d0840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.841419:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809280:192.168.202.132@tcp:41:rm.0 00800000:00000004:2.0:1788572482.841451:0:109280:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc4871c800: update ffff92fc700337d8 cache blocks 1802681/1916558 objects 523520/524085 00000080:00000004:2.0:1788572482.841459:0:109280:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc4871c800: update ffff92fc70031e18 cache blocks 1802681/1916558 objects 523520/524085 00000080:00000004:2.0:1788572482.841465:0:109280:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1802681/1916558 objects 523520/524085 00000080:00200000:2.0:1788572482.841644:0:109280:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8) pos/size0/4096 32bit_api 0 00000080:00200000:2.0:1788572482.841907:0:109280:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:2.0:1788572482.841936:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572482.841951:0:109280:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f59a.sanity-sec, flags=0 00000080:00200000:2.0:1788572482.841959:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572482.841964:0:109280:0:(namei.c:2351:ll_unlink()) VFS Op:name=f59a.sanity-sec, dir=[0x20000041c:0x1:0x0](ffff92fc665e63c8) 00010000:00010000:2.0:1788572482.842674:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572482.842814:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572482.842962:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc3e00/0x58a604dcef35210d lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x2:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221791e94 expref: -99 pid: 108851 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000002:00100000:2.0:1788572482.843018:0:109280:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67049c00 x1875449863806336/t4294983641(4294983641) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572482.843031:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572482.843037:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863809536/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572482.843062:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809536:192.168.202.132@tcp:35:rm.0 00000100:00100000:2.0:1788572482.843301:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d09c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.847622:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809536:192.168.202.132@tcp:35:rm.0 00000002:00100000:2.0:1788572482.847661:0:109280:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67049c00 x1875449863806336/t4294983641(4294983641) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572482.848230:0:109280:0:(vvp_object.c:117:vvp_conf_set()) [0x20000041c:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572482.848260:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788572482.848276:0:109280:0:(vvp_object.c:117:vvp_conf_set()) [0x20000041c:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572482.848317:0:109280:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc3e00/0x58a604dcef35210d lrc: 1/0,0 mode: --/CR res: [0x20000041c:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221791e94 expref: -99 pid: 108851 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572482.848325:0:109280:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 1/0,0 mode: --/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788572482.848334:0:109280:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 1/0,0 mode: --/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572482.848425:0:109280:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788572482.848429:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc51bc3e00/0x58a604dcef35210d lrc: 0/0,0 mode: --/CR res: [0x20000041c:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221791e94 expref: -99 pid: 108851 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572482.848444:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366e00/0x58a604dcef3520f1 lrc: 0/0,0 mode: --/PR res: [0x20000041c:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221791e47 expref: -99 pid: 108846 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788572482.848460:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef3520ea lrc: 0/0,0 mode: --/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791e2b expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:2.0:1788572482.848607:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572482.848618:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863809792/t0(0) o36->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 512/736 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572482.848647:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809792:192.168.202.132@tcp:36:rm.0 00000100:00100000:2.0:1788572482.848923:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d09c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.865043:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809792:192.168.202.132@tcp:36:rm.0 00000080:00200000:2.0:1788572482.865116:0:109280:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041c:0x2:0x0] ignore/verify layout 1/0, layout version 5 restore needed 0 00020000:00200000:2.0:1788572482.865128:0:109280:0:(lov_io.c:655:lov_io_mirror_init()) [0x20000041c:0x2:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:2.0:1788572482.865131:0:109280:0:(lov_io.c:709:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1788572482.865137:0:109280:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572482.865164:0:109280:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041c:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572482.865171:0:109280:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041c:0x2:0x0](ffff92fc665e2a08) 00000008:00010000:2.0:1788572482.865247:0:109280:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 5000 blocks 16 [cma]time: 1788572469 1788572464 1788572464 by oinfo size 5000 blocks 16 [cma]time 1788572469 1788572464 1788572475 ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 1/0,0 mode: PR/PR res: [0x280000401:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:2.0:1788572482.865294:0:109280:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 5000 blocks 16 [cma]time: 1788572469 1788572469 0 by oinfo size 5000 blocks 16 [cma]time 1788572469 1788572469 1788572476 ns: lustre-OST0001-osc-ffff92fc4871c800 lock: 000000004ea38ddf/0x58a604dcef35211b lrc: 1/0,0 mode: PR/PR res: [0x2c0000401:0x218:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221791ecc expref: -99 pid: 108855 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200008:2.0:1788572482.865365:0:109280:0:(file.c:479:ll_file_release()) START file d59a.sanity-sec:[0x20000041c:0x1:0x0](ffff92fc665e63c8), flags 704000 00010000:00010000:2.0:1788572482.865404:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572482.865448:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08),name=/ 00000002:00010000:2.0:1788572482.865477:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.865496:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572482.865506:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572482.865852:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.865862:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572482.865867:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) for lock 0x58a604dcef3520e3 00000080:00010000:2.0:1788572482.865877:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520e3 from it ffff9e0381d33c10 00010000:00010000:2.0:1788572482.865893:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572482.865904:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572482.865921:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572482.865941:0:109280:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59a.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665eec08), flags=0 00000080:00200000:2.0:1788572482.865946:0:109280:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59a.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665eec08), intent=getattr 00000002:00010000:2.0:1788572482.865961:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59a.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572482.866020:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.866028:0:109280:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.866040:0:109280:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109280 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572482.866065:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809920:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572482.866171:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d0180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.871652:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863809920:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572482.871677:0:109280:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871695:0:109280:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 4/1,0 mode: --/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871703:0:109280:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 4/1,0 mode: --/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871714:0:109280:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 4/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871742:0:109280:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 4/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871756:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 5/2,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871764:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 5/2,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871771:0:109280:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 4/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572482.871784:0:109280:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863809920/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572498 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572482.871796:0:109280:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871806:0:109280:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5deabc00 type 13 mode 4 res 8589935644/1 (0 0) 00000080:00200000:2.0:1788572482.871812:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572482.871834:0:109280:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041c:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1788572482.871837:0:109280:0:(namei.c:147:ll_iget()) got inode: [0x20000041c:0x1:0x0](ffff92fc665e63c8): rc = 0 00000080:00010000:2.0:1788572482.871847:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) for lock 0x58a604dcef352130 00000080:00010000:2.0:1788572482.871857:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041c:0x1:0x0](ffff92fc665e63c8) 00000080:00010000:2.0:1788572482.871861:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352130 from it ffff9e0381d33e00 00010000:00010000:2.0:1788572482.871864:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 3/1,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572482.871946:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572482.871980:0:109280:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08),name=/ 00000002:00010000:2.0:1788572482.872001:0:109280:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572482.872020:0:109280:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572482.872031:0:109280:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572482.872037:0:109280:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572482.872041:0:109280:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572482.872044:0:109280:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) for lock 0x58a604dcef3520e3 00000080:00010000:2.0:1788572482.872047:0:109280:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3520e3 from it ffff9e0381d33d88 00010000:00010000:2.0:1788572482.872050:0:109280:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572482.872057:0:109280:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572482.872065:0:109280:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572482.872072:0:109280:0:(namei.c:2261:ll_rmdir()) VFS Op:name=d59a.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665eec08) 00010000:00010000:2.0:1788572482.872089:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572482.872108:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 2/0,0 mode: PR/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572482.872120:0:109280:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 2/0,0 mode: CR/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1788572482.872264:0:109280:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617df80 x1875449863809152/t4294983646(4294983646) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572482.872276:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572482.872281:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863810176/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572482.872302:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863810176:192.168.202.132@tcp:35:rm.0 00000100:00100000:2.0:1788572482.872401:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d0180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.875799:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863810176:192.168.202.132@tcp:35:rm.0 00000002:00100000:2.0:1788572482.875837:0:109280:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617df80 x1875449863809152/t4294983646(4294983646) o101->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572498 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572482.876281:0:109280:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 1/0,0 mode: --/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572482.876292:0:109280:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 1/0,0 mode: --/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572482.876302:0:109280:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572482.876309:0:109280:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788572482.876313:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70367a00/0x58a604dcef352130 lrc: 0/0,0 mode: --/PR res: [0x20000041c:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791f2e expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572482.876332:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc70366000/0x58a604dcef352129 lrc: 0/0,0 mode: --/CR res: [0x20000041c:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791eef expref: -99 pid: 109280 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572482.876347:0:109280:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5600/0x58a604dcef3520e3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791e0f expref: -99 pid: 108846 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:2.0:1788572482.876366:0:109280:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4871c800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572482.876373:0:109280:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863810432/t0(0) o36->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 504/736 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572482.876404:0:109280:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863810432:192.168.202.132@tcp:36:rm.0 00000100:00100000:2.0:1788572482.876624:0:109280:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d0180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572482.883795:0:109280:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job rm:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109280:1875449863810432:192.168.202.132@tcp:36:rm.0 00000080:00200000:2.0:1788572482.883841:0:109280:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041c:0x1:0x0](ffff92fc665e63c8) 00000080:00200008:1.0:1788572483.495336:0:109295:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572483.495353:0:109295:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665eec08),name=/ 00000002:00010000:1.0:1788572483.495379:0:109295:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572483.495458:0:109295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5400/0x58a604dcef352137 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572483.495469:0:109295:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5400/0x58a604dcef352137 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572483.495482:0:109295:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5400/0x58a604dcef352137 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109295 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572483.495509:0:109295:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job lsof:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109295:1875449863810560:192.168.202.132@tcp:101:lsof.0 00000100:00100000:1.0:1788572483.495620:0:109295:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5a80 going to sleep for 11 seconds 00000080:00200000:1.0:1788572483.507158:0:109295:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572483.507166:0:109295:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) for lock 0x58a604dcef352137 00000080:00010000:1.0:1788572483.507172:0:109295:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665eec08) 00000080:00010000:1.0:1788572483.507176:0:109295:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352137 from it ffff9e03816abcb8 00010000:00010000:1.0:1788572483.507180:0:109295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5400/0x58a604dcef352137 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791f51 expref: -99 pid: 109295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572483.507189:0:109295:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4871c800 lock: ffff92fc702f5400/0x58a604dcef352137 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791f51 expref: -99 pid: 109295 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572485.124949:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c35a6bf6-0615-429a-bace-f01c5ef29804:2393:1875449863810688:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572485.125059:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2393:1875449863810944:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572485.125089:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2393:1875449863811200:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572485.125118:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572485.125124:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2393:1875449863810816:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572485.125145:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:2393:1875449863811072:192.168.202.132@tcp:400:kworker.0 00010000:00010000:2.0:1788572485.961089:0:108818:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572486.415091:0:108819:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572486.420270:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572486.420275:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4871c800: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572486.420281:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572486.420429:0:109303:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3520dc 00000020:00000080:2.0:1788572486.420447:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc69d5b800/LOV_OSC_UUID 00000020:00000080:1.0:1788572486.420591:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d5b800/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc4871c800 00000020:01000000:2.0:1788572486.420669:0:109303:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc4871c800 (flags='') 00000020:00000080:2.0:1788572486.420676:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572486.420686:0:109303:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc4871c800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572486.420690:0:109303:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc48d14d40) has no exports 00000008:01000000:2.0:1788572486.420971:0:109303:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc4871c800: client import never connected 00000100:00080000:2.0:1788572486.420975:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572486.420978:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4871c800: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572486.420981:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572486.421322:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572486.421330:0:109303:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc4871c800 (uuid e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00020000:01000000:2.0:1788572486.421335:0:109303:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc4871c800 00020000:01000000:2.0:1788572486.421337:0:109303:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc4871c800: disconnecting target lustre-OST0000-osc-ffff92fc4871c800 00000100:00080000:2.0:1788572486.421350:0:109303:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import e31c5b96-03bf-443d-89f4-f01a7ccb0fb7->lustre-OST0000_UUID 00010000:00010000:2.0:1788572486.421382:0:109303:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x226:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572486.421447:0:109303:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x226:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572486.421455:0:109303:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572486.421585:0:109303:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc460d0a80 going to sleep for 0 seconds 00000100:00100000:2.0:1788572486.421591:0:109303:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job umount:e31c5b96-03bf-443d-89f4-f01a7ccb0fb7:109303:1875449863811968:192.168.202.132@tcp:103:umount.0 00010000:00010000:2.0:1788572486.421599:0:109303:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572486.421604:0:109303:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572486.421608:0:109303:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc4871c800 lock: 00000000501eae9e/0x58a604dcef352106 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x226:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221791e78 expref: -99 pid: 108846 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00080000:2.0:1788572486.421697:0:109303:0:(import.c:65:import_set_state_nolock()) ffff92fc49237800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:1.0:1788572486.424653:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4536c800 for lustre-OST0001-osc-ffff92fc4871c800 00000020:00000080:1.0:1788572486.424678:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc45368800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000020:00000080:1.0:1788572486.424681:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc45368800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 for lustre-OST0001-osc-ffff92fc4871c800 00000020:01000000:1.0:1788572486.424692:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc4871c800 (e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000100:00080000:2.0:1788572486.436017:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572486.436024:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4871c800: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572486.436031:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572486.436172:0:109303:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3520d5 00000020:00000080:2.0:1788572486.436192:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc69d58000/LOV_OSC_UUID 00000020:01000000:2.0:1788572486.436251:0:109303:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc4871c800 (flags='') 00000020:00000080:2.0:1788572486.436258:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572486.436267:0:109303:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc4871c800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572486.436271:0:109303:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc48d119c0) has no exports 00000008:01000000:2.0:1788572486.437042:0:109303:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc4871c800: client import never connected 00000100:00080000:2.0:1788572486.437047:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572486.437051:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4871c800: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572486.437057:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572486.437284:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572486.437291:0:109303:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc4871c800 (uuid e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000020:00000080:2.0:1788572486.437297:0:109303:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3520ce 00000020:00000080:2.0:1788572486.437305:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc69d5c800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000100:00080000:1.0:1788572486.460274:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572486.460284:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4871c800: committing for last_committed 4294983643 gen 2 00000100:00100000:1.0:1788572486.460302:0:109303:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617d180 x1875449863809792/t4294983648(4294983648) o36->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572498 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572486.460342:0:109303:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617ea00 x1875449863810432/t4294983650(4294983650) o36->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 504/456 e 0 to 0 dl 1788572498 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572486.460355:0:109303:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6704a300 x1875449863812352/t4294983651(4294983651) o39->lustre-MDT0000-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572496 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572486.460371:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572486.462313:0:109303:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3520c0 00000020:00000080:1.0:1788572486.462332:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc69d0e800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000100:00080000:1.0:1788572486.463639:0:109303:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import e31c5b96-03bf-443d-89f4-f01a7ccb0fb7->lustre-MDT0001_UUID 00000100:00080000:1.0:1788572486.465294:0:109303:0:(import.c:65:import_set_state_nolock()) ffff92fc46150800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572486.469210:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572486.469215:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4871c800: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572486.469224:0:109303:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6728bb80 x1875449863812480/t4294976742(4294976742) o39->lustre-MDT0001-mdc-ffff92fc4871c800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572496 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572486.469246:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572486.473985:0:109303:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3520c7 00000020:00000080:1.0:1788572486.474002:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc69d09000/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000020:00000080:1.0:1788572486.474082:0:109303:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3520b9 00000020:00000080:1.0:1788572486.474087:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc69d0f000/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000020:01000000:1.0:1788572486.474095:0:109303:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc4871c800 (flags='') 00000020:00000080:1.0:1788572486.474104:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572486.474110:0:109303:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc4871c800: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572486.474114:0:109303:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc70034d40) has no exports 00000020:00000080:1.0:1788572486.474176:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572486.474181:0:109303:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc4871c800 (uuid e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000020:00000080:1.0:1788572486.474184:0:109303:0:(genops.c:885:class_export_put()) final put ffff92fc51c6f800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000020:00000080:1.0:1788572486.474187:0:109303:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51c6f800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 for lustre-clilov-ffff92fc4871c800 00000020:01000000:1.0:1788572486.474192:0:109303:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc4871c800 (e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000020:01000000:1.0:1788572486.474347:0:109303:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc4871c800 (flags='') 00000020:00000080:1.0:1788572486.474351:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572486.474354:0:109303:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc4871c800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572486.474356:0:109303:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc700319c0) has no exports 00000020:00000080:1.0:1788572486.474391:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572486.474394:0:109303:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc4871c800 (uuid e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000020:01000000:1.0:1788572486.474397:0:109303:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc4871c800 (flags='') 00000020:00000080:1.0:1788572486.474399:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572486.474401:0:109303:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc4871c800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572486.474403:0:109303:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc70033380) has no exports 00000008:01000000:1.0:1788572486.475396:0:109303:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc4871c800: client import never connected 00000100:00080000:1.0:1788572486.475402:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572486.475418:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4871c800: committing for last_committed 4294983643 gen 3 00000100:00100000:1.0:1788572486.475427:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572486.486161:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572486.486180:0:109303:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc4871c800 (uuid e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000020:01000000:1.0:1788572486.486186:0:109303:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc4871c800 (flags='') 00000020:00000080:1.0:1788572486.486192:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572486.486197:0:109303:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc4871c800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572486.486202:0:109303:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc48d13380) has no exports 00000020:00000080:2.0:1788572486.486746:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51c6a000 for lustre-MDT0000-mdc-ffff92fc4871c800 00000020:00000080:2.0:1788572486.486756:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc51c6e800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000020:00000080:2.0:1788572486.490914:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51c6e800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 for lustre-MDT0000-mdc-ffff92fc4871c800 00000020:01000000:2.0:1788572486.490934:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc4871c800 (e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000008:01000000:1.0:1788572486.493117:0:109303:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc4871c800: client import never connected 00000100:00080000:1.0:1788572486.493126:0:109303:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572486.493131:0:109303:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4871c800: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572486.493140:0:109303:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572486.497428:0:109303:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572486.497443:0:109303:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc4871c800 (uuid e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000080:02000400:1.0:1788572486.497447:0:109303:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572486.497450:0:109303:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1788572486.497501:0:109303:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc4871c800 00000100:00080000:1.0:1788572486.497510:0:109303:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import c35a6bf6-0615-429a-bace-f01c5ef29804->MGS 00010000:00010000:1.0:1788572486.497524:0:109303:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f4e00/0x58a604dcef352073 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221791dd0 expref: -99 pid: 108807 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572486.497538:0:109303:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f4e00/0x58a604dcef352073 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221791dd0 expref: -99 pid: 108807 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572486.497545:0:109303:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572486.497547:0:109303:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572486.497549:0:109303:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572486.497594:0:109303:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f4e00/0x58a604dcef352073 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791dd0 expref: -99 pid: 108807 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.497601:0:109303:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572486.497618:0:109303:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5480 going to sleep for 0 seconds 00000100:00100000:1.0:1788572486.497630:0:109303:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job umount:c35a6bf6-0615-429a-bace-f01c5ef29804:109303:1875449863812608:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572486.497639:0:109303:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572486.497647:0:109303:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572486.497650:0:109303:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f4e00/0x58a604dcef352073 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791dd0 expref: -99 pid: 108807 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572486.497676:0:109303:0:(import.c:65:import_set_state_nolock()) ffff92fc69d58800 MGS: changing import state from FULL to CONNECTING 00000020:00000080:1.0:1788572486.500162:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc46150800 for lustre-MDT0001-mdc-ffff92fc4871c800 00000020:00000080:1.0:1788572486.500181:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc46157800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 00000020:00000080:1.0:1788572486.500185:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46157800/e31c5b96-03bf-443d-89f4-f01a7ccb0fb7 for lustre-MDT0001-mdc-ffff92fc4871c800 00000020:01000000:1.0:1788572486.500194:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc4871c800 (e31c5b96-03bf-443d-89f4-f01a7ccb0fb7) 00000100:00100000:1.0:1788572486.531931:0:108808:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc53f75a00:108808] 0 exiting: rc = 1 00000100:00100000:1.0:1788572486.531991:0:108809:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc5dd61e00:108809] 1 exiting: rc = 0 00000100:00100000:2.0:1788572486.535001:0:108810:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc5dd60200:108810] 0 exiting: rc = 1 00000100:00100000:1.0:1788572486.548975:0:108813:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc51c13e00:108813] 1 exiting: rc = 0 00000100:00100000:2.0:1788572486.552577:0:108815:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc702bc200:108815] 0 exiting: rc = 1 00000100:00100000:2.0:1788572486.552686:0:108816:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc702bc400:108816] 1 exiting: rc = 0 00000100:00100000:2.0:1788572486.552758:0:108817:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc702bda00:108817] 2 exiting: rc = 0 00000100:00100000:2.0:1788572486.553115:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553130:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553136:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553141:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553147:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553158:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553164:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553169:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553174:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553185:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553192:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553199:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553204:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553209:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553214:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553218:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553230:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553236:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553241:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553246:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553251:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553263:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553269:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553274:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553279:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553290:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553296:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553302:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553307:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553312:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553318:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553329:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553335:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553340:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553345:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553351:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553363:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553369:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553374:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553379:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553390:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553396:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553404:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553424:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553429:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553435:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553441:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553446:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553452:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553464:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553471:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553477:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553482:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553494:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553499:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553504:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553516:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553522:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553527:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553533:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553544:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553550:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553555:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553567:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553575:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553580:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553586:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553599:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553605:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553616:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553623:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553629:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553641:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553648:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553653:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553659:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553669:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553675:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553680:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553685:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553689:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553701:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553706:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553712:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553752:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553759:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553764:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553776:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553782:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553788:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553793:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553804:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553811:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553816:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553822:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553834:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553841:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553846:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553851:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553856:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553869:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553876:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553881:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553887:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553898:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553905:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553910:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553915:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553928:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553934:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553941:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553946:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553958:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553964:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553969:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553975:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553981:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.553996:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554001:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554007:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554013:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554025:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554030:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554035:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554047:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554054:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554059:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572486.554064:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572486.716279:0:109316:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572486.716288:0:109316:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572486.716910:0:109316:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572486.717195:0:109316:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572486.717383:0:109316:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572486.717389:0:109316:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:1.0:1788572486.718439:0:109316:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:1.0:1788572486.720095:0:109316:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 10000000:01000000:2.0:1788572486.731861:0:109330:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000100:00100000:1.0:1788572486.791850:0:109316:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53c4bd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572486.797071:0:109316:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b3ef9aab-f8d6-453b-969f-24c8d2960d02:109316:1875449863813504:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572486.797522:0:109334:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b3ef9aab-f8d6-453b-969f-24c8d2960d02:109334:1875449863813632:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572486.797637:0:109334:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc451130c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572486.801510:0:109334:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b3ef9aab-f8d6-453b-969f-24c8d2960d02:109334:1875449863813632:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572486.801566:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572486.801581:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.801584:0:109334:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:1.0:1788572486.801591:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc69d44000 00000020:00000080:1.0:1788572486.801595:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572486.801607:0:109334:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc69d44000 (ffff92fc509e3380) 00000020:00000080:1.0:1788572486.801628:0:109334:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:1.0:1788572486.801635:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc69d44000 00000020:00000080:1.0:1788572486.801637:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:1.0:1788572486.801948:0:109334:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000000:1.0:1788572486.801953:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572486.801956:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.801958:0:109334:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:1.0:1788572486.801960:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572486.801963:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.801965:0:109334:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:1.0:1788572486.801969:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc69d44000 00000020:00000080:1.0:1788572486.801971:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572486.801977:0:109334:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc69d44000 (ffff92fc509e19c0) 00000020:00000080:1.0:1788572486.801986:0:109334:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:1.0:1788572486.801990:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc69d44000 00000020:00000080:1.0:1788572486.801992:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:1.0:1788572486.802154:0:109334:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000000:1.0:1788572486.802158:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572486.802162:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.802163:0:109334:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:1.0:1788572486.802166:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572486.802169:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.802170:0:109334:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:1.0:1788572486.802173:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:1.0:1788572486.802175:0:109334:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:1.0:1788572486.802178:0:109334:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:1.0:1788572486.802183:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572486.802185:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.802187:0:109334:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:1.0:1788572486.802190:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572486.802194:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.802195:0:109334:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:1.0:1788572486.802198:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572486.802201:0:109334:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572486.802223:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc69d44000 00000020:00000080:1.0:1788572486.802226:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572486.802232:0:109334:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc69d44000 (ffff92fc509e0000) 00000020:00000080:1.0:1788572486.802240:0:109334:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:1.0:1788572486.802245:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc69d44000 00000020:00000080:1.0:1788572486.802247:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572486.802305:0:109334:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5df7c800@lustre-MDT0000-mdc-ffff92fc69d44000: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572486.810676:0:109334:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc69d44000-1 is initialized 00000040:01000000:1.0:1788572486.810984:0:109334:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc69d44000 ctxt 13 is initialized 00000020:00000080:1.0:1788572486.814556:0:109334:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000000:1.0:1788572486.814697:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc69d44000 00000020:00000080:1.0:1788572486.814703:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:1.0:1788572486.814752:0:109334:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:1.0:1788572486.814806:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572486.814810:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.814812:0:109334:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:1.0:1788572486.814815:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:1.0:1788572486.814819:0:109334:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:1.0:1788572486.814820:0:109334:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572486.814822:0:109334:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572486.814824:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:1.0:1788572486.814826:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.814828:0:109334:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:1.0:1788572486.814830:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572486.814835:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.814837:0:109334:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:1.0:1788572486.814841:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572486.814848:0:109334:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572486.814888:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc69d44000 00000020:00000080:1.0:1788572486.814891:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572486.814904:0:109334:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc69d44000 (ffff92fc509e4d40) 00000020:00000080:1.0:1788572486.814925:0:109334:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:1.0:1788572486.814931:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc69d44000 00000020:00000080:1.0:1788572486.814934:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572486.815022:0:109334:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5df7d800@lustre-MDT0001-mdc-ffff92fc69d44000: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572486.825836:0:109334:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc69d44000-2 is initialized 00000040:01000000:1.0:1788572486.830417:0:109334:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc69d44000 ctxt 13 is initialized 00000020:00000080:1.0:1788572486.842508:0:109334:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000000:1.0:1788572486.842538:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc69d44000 00000020:00000080:1.0:1788572486.842544:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:1.0:1788572486.842588:0:109334:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:1.0:1788572486.842626:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572486.842630:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.842632:0:109334:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:1.0:1788572486.842636:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:1.0:1788572486.842639:0:109334:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:1.0:1788572486.842641:0:109334:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572486.842643:0:109334:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572486.842645:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:1.0:1788572486.842647:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.842648:0:109334:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:1.0:1788572486.842651:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:1.0:1788572486.842656:0:109334:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:1.0:1788572486.842660:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572486.842661:0:109334:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:1.0:1788572486.842664:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572486.842670:0:109334:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572486.842699:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc69d44000 00000020:00000080:1.0:1788572486.842702:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572486.842714:0:109334:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc69d44000 (ffff92fc49930000) 00000020:00000080:1.0:1788572486.842748:0:109334:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:1.0:1788572486.842755:0:109334:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc69d44000 00000020:00000080:1.0:1788572486.842757:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572486.846074:0:109334:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5df7e000@lustre-OST0000-osc-ffff92fc69d44000: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572486.846319:0:109334:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc69d44000-3 is initialized 00000100:00100000:2.0:1788572486.870770:0:109334:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b3ef9aab-f8d6-453b-969f-24c8d2960d02:109334:1875449863813760:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572486.870815:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572486.870828:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572486.870830:0:109334:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572486.870836:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572486.870880:0:109334:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b3ef9aab-f8d6-453b-969f-24c8d2960d02:109334:1875449863813888:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572486.870970:0:109334:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc50c84e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572486.879692:0:109334:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b3ef9aab-f8d6-453b-969f-24c8d2960d02:109334:1875449863813888:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572486.879934:0:109334:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572486.879944:0:109334:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572486.879946:0:109334:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:2.0:1788572486.879952:0:109334:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:1.0:1788572486.951942:0:109316:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:1.0:1788572486.951973:0:109316:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788572486.951993:0:109316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.952006:0:109316:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788572486.952028:0:109316:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1788572486.952053:0:109316:0:(genops.c:1385:class_connect()) connect: client 2ec24f33-d5f0-4af1-8380-11f8d2549b2a, cookie 0x58a604dcef352192 00800000:01000000:1.0:1788572486.952079:0:109316:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 2ec24f33-d5f0-4af1-8380-11f8d2549b2a to lustre-clilmv-ffff92fc69d44000 00800000:01000000:1.0:1788572486.952087:0:109316:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc69d44000(2ec24f33-d5f0-4af1-8380-11f8d2549b2a) - lustre-MDT0000_UUID, 2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:00000080:1.0:1788572486.952095:0:109316:0:(genops.c:1385:class_connect()) connect: client 2ec24f33-d5f0-4af1-8380-11f8d2549b2a, cookie 0x58a604dcef352199 00000100:00080000:1.0:1788572486.952106:0:109316:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc69d44000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572486.952121:0:109316:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc69d44000: import ffff92fc5df7c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572486.952137:0:109316:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572486.952175:0:109316:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449863814912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572486.952196:0:109316:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-MDT0000_UUID 00800000:01000000:1.0:1788572486.952289:0:109316:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc69d44000(2ec24f33-d5f0-4af1-8380-11f8d2549b2a) successfully (5) 00800000:00000004:1.0:1788572486.952292:0:109316:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc69d44000: age 3242, max_age 0 00800000:01000000:1.0:1788572486.952322:0:109316:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc69d44000(2ec24f33-d5f0-4af1-8380-11f8d2549b2a) - lustre-MDT0001_UUID, 2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:00000080:1.0:1788572486.952329:0:109316:0:(genops.c:1385:class_connect()) connect: client 2ec24f33-d5f0-4af1-8380-11f8d2549b2a, cookie 0x58a604dcef3521a0 00000100:00080000:1.0:1788572486.952334:0:109316:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc69d44000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572486.952339:0:109316:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc69d44000: import ffff92fc5df7d800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572486.952345:0:109316:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572486.952356:0:109316:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449863815168/t0(0) o38->lustre-MDT0001-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572486.952365:0:109316:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-MDT0001_UUID 00800000:01000000:1.0:1788572486.952405:0:109316:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc69d44000(2ec24f33-d5f0-4af1-8380-11f8d2549b2a) successfully (5) 00800000:00000004:1.0:1788572486.952419:0:109316:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc69d44000: age 3242, max_age 0 00000080:00000004:1.0:1788572486.952440:0:109316:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc69d44000: age 3242, max_age 4241 00800000:00000004:1.0:1788572486.952446:0:109316:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc69d44000: age 3242, max_age 4241 00000100:00080000:1.0:1788572486.952465:0:109316:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67288000 x1875449863815424/t0(0) o41->lustre-MDT0000-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572486.952474:0:109316:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70656b40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572486.952510:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67289880 x1875449863815040/t0(0) o41->lustre-MDT0000-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572486.952521:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:1.0:1788572486.952526:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67288e00 x1875449863815296/t0(0) o41->lustre-MDT0001-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572486.963292:0:109316:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70656b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572486.964076:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:2392:1875449863815168:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572486.972679:0:109316:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863815424:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:2.0:1788572486.972761:0:109316:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc69d44000: update ffff92fc509e0458 cache blocks 317013/353529 objects 1023703/1024000 00000080:00000004:2.0:1788572486.972771:0:109316:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc69d44000: update ffff92fc509e1e18 cache blocks 317013/353529 objects 1023703/1024000 00000100:00100000:2.0:1788572486.972827:0:109316:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863815552:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:2.0:1788572486.972908:0:109316:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc50c84600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572486.973004:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:0:1875449863815296:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572486.973019:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:0:1875449863815040:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572486.983039:0:109316:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863815552:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:2.0:1788572486.983076:0:109316:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:2.0:1788572486.983091:0:109316:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:2.0:1788572486.983097:0:109316:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:2.0:1788572486.983115:0:109316:0:(genops.c:1385:class_connect()) connect: client 2ec24f33-d5f0-4af1-8380-11f8d2549b2a, cookie 0x58a604dcef3521a7 00000020:00000080:2.0:1788572486.983136:0:109316:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef3521ae 00000100:00080000:2.0:1788572486.983144:0:109316:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc69d44000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572486.983153:0:109316:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc69d44000: import ffff92fc5df7e000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572486.983166:0:109316:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572486.983197:0:109316:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449863815680/t0(0) o8->lustre-OST0000-osc-ffff92fc69d44000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572486.983258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:2392:1875449863815680:192.168.202.132@tcp:8:mount.lustre.0 00000100:00080000:2.0:1788572486.983640:0:109316:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-OST0000_UUID 00020000:01000000:2.0:1788572486.983711:0:109316:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc69d44000) active 00000020:00000080:2.0:1788572486.983768:0:109316:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef3521b5 00000100:00080000:2.0:1788572486.983774:0:109316:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc69d44000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572486.983779:0:109316:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc69d44000: import ffff92fc51c6a800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572486.983787:0:109316:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572486.983801:0:109316:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449863815808/t0(0) o8->lustre-OST0001-osc-ffff92fc69d44000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572486.986871:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:2392:1875449863815808:192.168.202.132@tcp:8:mount.lustre.0 00000100:00080000:2.0:1788572486.992111:0:109316:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-OST0001_UUID 00020000:01000000:2.0:1788572486.992131:0:109316:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc69d44000) active 00000100:00100000:2.0:1788572486.992222:0:109316:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863815936:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788572486.992311:0:109316:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc50c84600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572486.996525:0:109316:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863815936:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788572486.996600:0:109316:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863816064:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:2.0:1788572486.996771:0:109316:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc50c84600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572487.013607:0:109316:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109316:1875449863816064:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:2.0:1788572487.013836:0:109316:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e63c8) 00000080:00200000:2.0:1788572487.013846:0:109316:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:2.0:1788572487.014277:0:109316:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:2.0:1788572487.014282:0:109316:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00010000:00010000:1.0:1788572490.823554:0:109327:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572491.551157:0:109520:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-sec test 59b: migrate/extend/split of encrypted files without key ========================================================== 21:41:28 (1788572488) 00010000:00010000:2.0:1788572491.847485:0:109328:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572492.868938:0:109327:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572492.878493:0:109552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job lsof:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109552:1875449863816832:192.168.202.132@tcp:101:lsof.0 00010000:00010000:1.0:1788572492.878522:0:109552:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878536:0:109552:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878549:0:109552:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878568:0:109552:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878580:0:109552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878588:0:109552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878594:0:109552:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572492.878605:0:109552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704b100 x1875449863816832/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572508 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572492.878619:0:109552:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878631:0:109552:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f00000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572492.878638:0:109552:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572492.878672:0:109552:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572492.878677:0:109552:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e63c8) for lock 0x58a604dcef3521bc 00000080:00010000:1.0:1788572492.878682:0:109552:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e63c8) 00000080:00010000:1.0:1788572492.878686:0:109552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3521bc from it ffff9e0380c97cb8 00010000:00010000:1.0:1788572492.878690:0:109552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572492.878699:0:109552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572493.893628:0:109328:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572494.917916:0:109327:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572494.959014:0:109561:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e63c8) 00000080:00200000:2.0:1788572494.959050:0:109561:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc69d44000 (sb ffff92fc69d44000) 00010000:00010000:2.0:1788572494.959087:0:109561:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef352153 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791f66 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959099:0:109561:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef352153 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791f66 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959106:0:109561:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef352153 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791f66 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959113:0:109561:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef352153 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791f66 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959186:0:109561:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572494.959194:0:109561:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3800/0x58a604dcef352184 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791f6d expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959201:0:109561:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3800/0x58a604dcef352184 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791f6d expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959207:0:109561:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3800/0x58a604dcef352184 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791f6d expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.959213:0:109561:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3800/0x58a604dcef352184 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791f6d expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572494.959873:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3ef9aab-f8d6-453b-969f-24c8d2960d02:2393:1875449863816960:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572494.959997:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572494.960004:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3ef9aab-f8d6-453b-969f-24c8d2960d02:2393:1875449863817088:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572494.960740:0:109561:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:2.0:1788572494.960756:0:109561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3800/0x58a604dcef352184 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791f6d expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572494.960776:0:109561:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:2.0:1788572494.960786:0:109561:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.960794:0:109561:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.960801:0:109561:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572494.960808:0:109561:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572494.968240:0:109327:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572494.968256:0:109327:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572494.968269:0:109327:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2200 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572494.968274:0:109327:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572494.968285:0:109327:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572494.968292:0:109327:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572494.968298:0:109327:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572494.968301:0:109327:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 00010000:00010000:1.0:1788572494.968339:0:109327:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 3/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572494.968351:0:109327:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572494.968471:0:109327:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572494.968481:0:109327:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572494.968483:0:109327:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572494.968485:0:109327:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572494.968493:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572494.968516:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b3ef9aab-f8d6-453b-969f-24c8d2960d02:2394:1875449863817216:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572494.969027:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3ef9aab-f8d6-453b-969f-24c8d2960d02:2393:1875449863817088:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572494.969038:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b3ef9aab-f8d6-453b-969f-24c8d2960d02:2393:1875449863816960:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572494.969167:0:109561:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:2.0:1788572494.969177:0:109561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef35218b lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791f74 expref: -99 pid: 109316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572494.969196:0:109561:0:(mgc_request.c:130:config_log_put()) dropping config log params 10000000:01000000:2.0:1788572494.969201:0:109561:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:2.0:1788572494.969204:0:109561:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:2.0:1788572494.969817:0:109561:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc58333000 active: 1 00020000:01000000:2.0:1788572494.969822:0:109561:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc58332800 active: 1 00020000:01000000:2.0:1788572494.969824:0:109561:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788572494.969829:0:109561:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc69d44000 00020000:01000000:2.0:1788572494.969831:0:109561:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc69d44000: disconnecting target lustre-OST0001-osc-ffff92fc69d44000 00000100:00080000:2.0:1788572494.969852:0:109561:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-OST0001_UUID 00000100:00080000:2.0:1788572494.970431:0:109561:0:(import.c:65:import_set_state_nolock()) ffff92fc51c6a800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:1.0:1788572494.970520:0:109328:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572494.988253:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572494.988258:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc69d44000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572494.988263:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572494.988500:0:109561:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3521b5 00000020:00000080:2.0:1788572494.988572:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc58332800/LOV_OSC_UUID 00000020:01000000:2.0:1788572494.988654:0:109561:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc69d44000 (flags='') 00000020:00000080:2.0:1788572494.988659:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572494.988680:0:109561:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc69d44000: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572494.988685:0:109561:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc48cc8000) has no exports 00000008:01000000:2.0:1788572494.990047:0:109561:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc69d44000: client import never connected 00000100:00080000:2.0:1788572494.990051:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572494.990055:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc69d44000: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572494.990060:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572494.990222:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572494.990231:0:109561:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00020000:01000000:2.0:1788572494.990236:0:109561:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc69d44000 00020000:01000000:2.0:1788572494.990238:0:109561:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc69d44000: disconnecting target lustre-OST0000-osc-ffff92fc69d44000 00000100:00080000:2.0:1788572494.990252:0:109561:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-OST0000_UUID 00000100:00080000:2.0:1788572494.990952:0:109561:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7e000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572494.991014:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:2392:1875449863817472:192.168.202.132@tcp:9:umount.0 00000100:00080000:2.0:1788572494.993350:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572494.993364:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc69d44000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572494.993370:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572494.993533:0:109561:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3521ae 00000020:00000080:2.0:1788572494.993555:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc58333000/LOV_OSC_UUID 00000020:01000000:2.0:1788572494.993613:0:109561:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc69d44000 (flags='') 00000020:00000080:2.0:1788572494.993619:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572494.993629:0:109561:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc69d44000: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572494.993634:0:109561:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc49930000) has no exports 00000008:01000000:2.0:1788572494.994132:0:109561:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc69d44000: client import never connected 00000100:00080000:2.0:1788572494.994135:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572494.994138:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc69d44000: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572494.994141:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572494.994346:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572494.994352:0:109561:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:00000080:2.0:1788572494.994357:0:109561:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3521a7 00000020:00000080:2.0:1788572494.994363:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc58332000/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000100:00080000:2.0:1788572494.995628:0:109561:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-MDT0000_UUID 00010000:00010000:2.0:1788572494.995790:0:109561:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572494.996072:0:109561:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572494.996085:0:109561:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572494.996101:0:109561:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc433fe840 going to sleep for 0 seconds 00000100:00100000:2.0:1788572494.996111:0:109561:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job umount:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:109561:1875449863817600:192.168.202.132@tcp:103:umount.0 00010000:00010000:2.0:1788572494.996121:0:109561:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572494.996128:0:109561:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572494.996131:0:109561:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d44000 lock: ffff92fc51bc2400/0x58a604dcef3521bc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791fac expref: -99 pid: 109552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00080000:2.0:1788572494.998005:0:109561:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7c800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:2.0:1788572495.002371:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572495.002376:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc69d44000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572495.002393:0:109561:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617c380 x1875449863817728/t4294983652(4294983652) o39->lustre-MDT0000-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572504 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572495.002417:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572495.004405:0:109561:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352199 00000020:00000080:2.0:1788572495.004419:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc5df7c000/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000100:00080000:2.0:1788572495.004514:0:109561:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2ec24f33-d5f0-4af1-8380-11f8d2549b2a->lustre-MDT0001_UUID 00000100:00080000:2.0:1788572495.008324:0:109561:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7d800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572495.009894:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2ec24f33-d5f0-4af1-8380-11f8d2549b2a:2392:1875449863817856:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572495.014858:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:2.0:1788572495.014863:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc69d44000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572495.014873:0:109561:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617c380 x1875449863817856/t4294976743(4294976743) o39->lustre-MDT0001-mdc-ffff92fc69d44000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572505 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572495.014897:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572495.019444:0:109561:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3521a0 00000020:00000080:2.0:1788572495.019463:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc5df7d000/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:00000080:2.0:1788572495.021244:0:109561:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352192 00000020:00000080:2.0:1788572495.021253:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc5df78000/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:01000000:2.0:1788572495.021267:0:109561:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc69d44000 (flags='') 00000020:00000080:2.0:1788572495.021276:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572495.021282:0:109561:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc69d44000: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572495.021287:0:109561:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc509e3380) has no exports 00000020:00000080:2.0:1788572495.021416:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572495.021423:0:109561:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:00000080:2.0:1788572495.021426:0:109561:0:(genops.c:885:class_export_put()) final put ffff92fc5df7e800/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:00000080:2.0:1788572495.021429:0:109561:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df7e800/2ec24f33-d5f0-4af1-8380-11f8d2549b2a for lustre-clilov-ffff92fc69d44000 00000020:01000000:2.0:1788572495.021434:0:109561:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc69d44000 (2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000000:2.0:1788572495.027751:0:109561:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc69d44000 (flags='') 00000020:00000080:2.0:1788572495.027766:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572495.027775:0:109561:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc69d44000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572495.027780:0:109561:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc509e19c0) has no exports 00000020:00000080:2.0:1788572495.027838:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572495.027845:0:109561:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc69d44000 (uuid 2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000000:2.0:1788572495.027849:0:109561:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc69d44000 (flags='') 00000020:00000080:2.0:1788572495.027853:0:109561:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572495.027856:0:109561:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc69d44000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572495.027858:0:109561:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc509e0000) has no exports 00000008:01000000:2.0:1788572495.030013:0:109561:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc69d44000: client import never connected 00000100:00080000:2.0:1788572495.030021:0:109561:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572495.030026:0:109561:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc69d44000: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572495.030035:0:109561:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572495.047085:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5df7c800 for lustre-MDT0000-mdc-ffff92fc69d44000 00000020:00000080:2.0:1788572495.047099:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc5df7a800/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:00000080:2.0:1788572495.047102:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df7a800/2ec24f33-d5f0-4af1-8380-11f8d2549b2a for lustre-MDT0000-mdc-ffff92fc69d44000 00000020:01000000:2.0:1788572495.047114:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc69d44000 (2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:00000080:1.0:1788572495.058447:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5df7d800 for lustre-MDT0001-mdc-ffff92fc69d44000 00000020:00000080:1.0:1788572495.058460:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc5df7b000/2ec24f33-d5f0-4af1-8380-11f8d2549b2a 00000020:00000080:1.0:1788572495.058464:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df7b000/2ec24f33-d5f0-4af1-8380-11f8d2549b2a for lustre-MDT0001-mdc-ffff92fc69d44000 00000020:01000000:1.0:1788572495.058478:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc69d44000 (2ec24f33-d5f0-4af1-8380-11f8d2549b2a) 00000020:01000004:1.0:1788572495.266449:0:109573:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:1.0:1788572495.266453:0:109573:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:1.0:1788572495.266455:0:109573:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:1.0:1788572495.266456:0:109573:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock,test_dummy_encryption 00000080:01200004:1.0:1788572495.266564:0:109573:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc492a7800 00000080:01000004:1.0:1788572495.266573:0:109573:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:1.0:1788572495.266591:0:109573:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:1.0:1788572495.266624:0:109573:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:1.0:1788572495.266639:0:109573:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc499319c0) 00000020:00000080:1.0:1788572495.266654:0:109573:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:1.0:1788572495.266939:0:109573:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572495.266962:0:109573:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572495.267099:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:1.0:1788572495.267105:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572495.267901:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572495.267910:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:1.0:1788572495.268273:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:1.0:1788572495.268280:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572495.268615:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572495.268622:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:2.0:1788572495.273370:0:109573:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:2.0:1788572495.273667:0:109573:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:2.0:1788572495.274154:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:2.0:1788572495.274160:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788572495.274656:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788572495.274662:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:2.0:1788572495.275152:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:2.0:1788572495.275159:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788572495.275503:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788572495.275509:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:1.0:1788572495.276645:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:1.0:1788572495.276658:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:1.0:1788572495.277511:0:109573:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:1.0:1788572495.277519:0:109573:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 10000000:01000000:2.0:1788572495.285047:0:109587:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000100:00100000:2.0:1788572495.336503:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863818624:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572495.336534:0:109573:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.336547:0:109573:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572495.336628:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863818752:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572495.336713:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5dda4600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.342188:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863818752:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572495.342252:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863818880:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572495.342353:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5dda4600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.352125:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863818880:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572495.353410:0:109592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109592:1875449863819008:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572495.353513:0:109592:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5dda4b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.360093:0:109592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109592:1875449863819008:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572495.360151:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.360165:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.360167:0:109592:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1788572495.360175:0:109592:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc492a7800 00000020:00000080:2.0:1788572495.360177:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572495.360191:0:109592:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc492a7800 (ffff92fc48d10000) 00000020:00000080:2.0:1788572495.360213:0:109592:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:2.0:1788572495.360219:0:109592:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc492a7800 00000020:00000080:2.0:1788572495.360222:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572495.360497:0:109592:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:01000000:2.0:1788572495.360502:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572495.360506:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.360508:0:109592:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1788572495.360511:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.360514:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.360515:0:109592:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572495.360519:0:109592:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc492a7800 00000020:00000080:2.0:1788572495.360521:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572495.360528:0:109592:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc492a7800 (ffff92fc48d13380) 00000020:00000080:2.0:1788572495.360537:0:109592:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:2.0:1788572495.360541:0:109592:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc492a7800 00000020:00000080:2.0:1788572495.360543:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572495.360699:0:109592:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:01000000:2.0:1788572495.360703:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572495.360706:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.360707:0:109592:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572495.360709:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.360712:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.360713:0:109592:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1788572495.360716:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1788572495.360719:0:109592:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1788572495.362605:0:109592:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1788572495.362624:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572495.362631:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.362633:0:109592:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1788572495.362637:0:109592:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.362643:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.362645:0:109592:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1788572495.362648:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572495.362654:0:109592:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572495.362682:0:109592:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc492a7800 00000020:00000080:2.0:1788572495.362685:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572495.362697:0:109592:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc492a7800 (ffff92fc48d119c0) 00000020:00000080:2.0:1788572495.362714:0:109592:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:2.0:1788572495.362735:0:109592:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc492a7800 00000020:00000080:2.0:1788572495.362738:0:109592:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572495.365116:0:109592:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc69d47000@lustre-MDT0000-mdc-ffff92fc492a7800: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572495.375377:0:109592:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc492a7800-1 is initialized 00000020:01000000:2.0:1788572495.425655:0:109573:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:2.0:1788572495.426117:0:109573:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572495.426180:0:109573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.426194:0:109573:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572495.426204:0:109573:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc492a7800 from 1 10000000:01000000:2.0:1788572495.426207:0:109573:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572495.426256:0:109573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.426270:0:109573:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.426283:0:109573:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572495.426318:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819392:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572495.426429:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.436069:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819392:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572495.436107:0:109573:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.436119:0:109573:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572495.436868:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819520:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572495.437063:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.444678:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819520:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572495.445463:0:109573:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572495.445496:0:109573:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.445509:0:109573:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572495.446100:0:109573:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc492a7800 from 1 10000000:01000000:2.0:1788572495.446104:0:109573:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572495.446248:0:109573:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.446257:0:109573:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.446268:0:109573:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572495.446294:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819648:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572495.446379:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.454899:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819648:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572495.454932:0:109573:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.454945:0:109573:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572495.455035:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819776:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572495.455377:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.458698:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819776:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572495.458961:0:109573:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819904:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572495.459089:0:109573:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.462330:0:109573:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109573:1875449863819904:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572495.462893:0:109599:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109599:1875449863820032:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572495.463261:0:109599:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.468195:0:109599:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109599:1875449863820032:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572495.468249:0:109599:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.468262:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.468265:0:109599:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572495.468272:0:109599:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc492a7800 00000020:00000080:2.0:1788572495.468275:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572495.468351:0:109599:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572495.468356:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.468358:0:109599:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572495.468361:0:109599:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.468363:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.468364:0:109599:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572495.468367:0:109599:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc492a7800 00000020:00000080:2.0:1788572495.468369:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572495.468678:0:109599:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572495.468683:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.468685:0:109599:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:2.0:1788572495.468687:0:109599:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572495.468690:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572495.468692:0:109599:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:2.0:1788572495.468695:0:109599:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc492a7800 00000020:00000080:2.0:1788572495.468697:0:109599:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:2.0:1788572495.469438:0:109599:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 734us 00000100:00100000:2.0:1788572495.469596:0:109599:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109599:1875449863820160:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572495.469712:0:109599:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.477449:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2392:1875449863820288:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572495.477548:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2392:1875449863820544:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572495.503792:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:0:1875449863820672:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572495.504195:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:0:1875449863820416:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572495.521966:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc492a7800: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572495.521980:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc492a7800: connect to target with instance 1 00000100:00080000:1.0:1788572495.521989:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc492a7800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572495.522002:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788572495.522005:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46150000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788572495.522012:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:1.0:1788572495.522017:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572495.522021:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788572495.522037:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2392:1875449863821056:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572495.522055:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc492a7800: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572495.522060:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc492a7800: connect to target with instance 1 00000100:00080000:1.0:1788572495.522064:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc492a7800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572495.522068:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572495.522070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46153800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788572495.522074:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:1.0:1788572495.522076:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572495.522079:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788572495.522083:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2392:1875449863821184:192.168.202.132@tcp:8:mount.lustre.0 00000080:00200000:2.0:1788572495.688622:0:109617:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8),name=/ 00000002:00010000:2.0:1788572495.688657:0:109617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572495.688766:0:109617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.688778:0:109617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.688790:0:109617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572495.688817:0:109617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109617:1875449863821568:192.168.202.132@tcp:101:mkdir.0 00000100:00100000:2.0:1788572495.688903:0:109617:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.692711:0:109617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109617:1875449863821568:192.168.202.132@tcp:101:mkdir.0 00010000:00010000:2.0:1788572495.692805:0:109617:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692816:0:109617:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692828:0:109617:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692842:0:109617:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692860:0:109617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692868:0:109617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692876:0:109617:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572495.692889:0:109617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ce00 x1875449863821568/t0(0) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572511 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572495.692904:0:109617:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692917:0:109617:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc53f87c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572495.692924:0:109617:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572495.692953:0:109617:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572495.692958:0:109617:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) for lock 0x58a604dcef352241 00000080:00010000:2.0:1788572495.692964:0:109617:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00010000:2.0:1788572495.692968:0:109617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352241 from it ffff9e0381ec3be0 00010000:00010000:2.0:1788572495.692971:0:109617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572495.692981:0:109617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572495.693012:0:109617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572495.693036:0:109617:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665d80c8), flags=1536 00000080:00200000:2.0:1788572495.693041:0:109617:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8),name=/ 00000002:00010000:2.0:1788572495.693057:0:109617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572495.693065:0:109617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572495.693077:0:109617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572495.693083:0:109617:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572495.693087:0:109617:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00010000:2.0:1788572495.693090:0:109617:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) for lock 0x58a604dcef352241 00000080:00010000:2.0:1788572495.693094:0:109617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352241 from it ffff9e0381ec3ce0 00010000:00010000:2.0:1788572495.693096:0:109617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572495.693104:0:109617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572495.693111:0:109617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572495.693117:0:109617:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8),name=/ 00000002:00010000:2.0:1788572495.693122:0:109617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572495.693128:0:109617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572495.693142:0:109617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572495.693149:0:109617:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572495.693152:0:109617:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00010000:2.0:1788572495.693155:0:109617:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) for lock 0x58a604dcef352241 00000080:00010000:2.0:1788572495.693158:0:109617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352241 from it ffff9e0381ec3dc8 00010000:00010000:2.0:1788572495.693162:0:109617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572495.693169:0:109617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572495.693459:0:109617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572495.693467:0:109617:0:(namei.c:2172:do_mkdir()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665d80c8) 00000100:00100000:2.0:1788572495.726552:0:109617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109617:1875449863821696:192.168.202.132@tcp:700:mkdir.0 00000100:00100000:2.0:1788572495.726700:0:109617:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.730462:0:109617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109617:1875449863821696:192.168.202.132@tcp:700:mkdir.0 00010000:00010000:2.0:1788572495.730515:0:109617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572495.730581:0:109617:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572495.730589:0:109617:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572495.730592:0:109617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352241 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221791ff9 expref: -99 pid: 109617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000020:00100000:2.0:1788572495.730621:0:109617:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc492a7800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572495.730629:0:109617:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ce00 x1875449863821952/t0(0) o36->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 592/688 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572495.730652:0:109617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109617:1875449863821952:192.168.202.132@tcp:36:mkdir.0 00000100:00100000:2.0:1788572495.730861:0:109617:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572495.741521:0:109617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mkdir:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109617:1875449863821952:192.168.202.132@tcp:36:mkdir.0 00000080:00200000:2.0:1788572495.741571:0:109617:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e63c8) 00000080:00200000:2.0:1788572495.741580:0:109617:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000100:00100000:1.0:1788572496.070082:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:2.0:1788572496.075160:0:109626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863822848:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572496.075186:0:109626:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075200:0:109626:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075207:0:109626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075214:0:109626:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075222:0:109626:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075228:0:109626:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204600 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572496.075234:0:109626:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075240:0:109626:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075247:0:109626:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075254:0:109626:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572496.075261:0:109626:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.075267:0:109626:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204600/0x58a604dcef35225d lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572496.075286:0:109626:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff92fc67339500 x1875449863822848/t4294983655(4294983655) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572496.075299:0:109626:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67339500 x1875449863822848/t4294983655(4294983655) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572496.075317:0:109626:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572496.075427:0:109626:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572496.075436:0:109626:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572496.075479:0:109626:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 0 00000002:00100000:2.0:1788572496.075498:0:109626:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67339500 x1875449863822848/t4294983655(4294983655) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572496.075543:0:109626:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67339500 x1875449863822848/t4294983655(4294983655) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572496.075552:0:109626:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc492a7800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572496.075557:0:109626:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863823232/t0(0) o35->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572496.075581:0:109626:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863823232:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572496.075652:0:109626:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.078958:0:109626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863823232:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572496.079005:0:109626:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67339500 x1875449863822848/t4294983655(4294983655) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572512 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572496.079042:0:109626:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000041d:0x2:0x0](ffff92fc665db248) 00000002:00010000:2.0:1788572496.079053:0:109626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572496.079110:0:109626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.079120:0:109626:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.079143:0:109626:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109626 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572496.079164:0:109626:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863823360:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572496.079238:0:109626:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.082077:0:109626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863823360:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572496.082109:0:109626:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.082134:0:109626:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572496.082157:0:109626:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff92fc4617d180 x1875449863823360/t0(0) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572512 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572496.082167:0:109626:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572496.082181:0:109626:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.082195:0:109626:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc53f87c00 type 13 mode 16 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572496.082202:0:109626:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572496.082223:0:109626:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665db248) for lock 0x58a604dcef352264 00000080:00010000:2.0:1788572496.082231:0:109626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352264 from it ffff9e0382303c38 00010000:00010000:2.0:1788572496.082236:0:109626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.082247:0:109626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572496.082273:0:109626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572496.082285:0:109626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572496.082294:0:109626:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000041d:0x2:0x0](00000000cfbfb02b) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572496.082337:0:109626:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788572496.082341:0:109626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572496.082349:0:109626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572496.082369:0:109626:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572496.082393:0:109626:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572496.087595:0:109626:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572496.087641:0:109626:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665db248), flags 100001 00000002:00100000:2.0:1788572496.087999:0:109626:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67338700 x1875449863822592/t4294983654(4294983654) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 568/640 e 0 to 0 dl 1788572512 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572496.088089:0:109626:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc492a7800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572496.088096:0:109626:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863823744/t0(0) o35->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572496.088119:0:109626:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863823744:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572496.088313:0:109626:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.093411:0:109626:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc492a7800: committing for last_committed 4294983657 gen 1 00000100:00100000:2.0:1788572496.093501:0:109626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109626:1875449863823744:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572496.093538:0:109626:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67338700 x1875449863822592/t4294983654(4294983654) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 568/640 e 0 to 0 dl 1788572512 ref 1 fl Complete:RMQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572496.128291:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2394:1875449863824640:192.168.202.132@tcp:10:dd.0 00000080:00200000:1.0:1788572496.131644:0:109627:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 0 i_blocks 0 00000020:00010000:1.0:1788572496.131652:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c308 (ffff92fc58255ab8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572496.131656:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c308 (ffff92fc58255ab8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572496.131661:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2bb0 (ffff92fc58255368/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572496.131671:0:109627:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572496.131687:0:109627:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788572496.131709:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c308 (ffff92fc58255ab8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572496.131713:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2bb0 (ffff92fc58255ab8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572496.131769:0:109627:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788572496.131786:0:109627:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665db248) current size 0, valid attrs 1000a168, mode 0, uid 2, gid 0, new size 0, atime 1788572496.000000000, mtime 1788572496.000000000, ctime 1788572496.000000000, rc 0 00000080:00200008:1.0:1788572496.132065:0:109627:0:(file.c:2650:do_file_write_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 0, count: 9000 00000080:00200000:1.0:1788572496.132075:0:109627:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: write ppos: 0, bytes: 9000 00000080:00200000:1.0:1788572496.132080:0:109627:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 0 00000020:00200000:1.0:1788572496.132086:0:109627:0:(cl_io.c:214:cl_io_rw_init()) header@00000000301160e7[0x110, 1, [0x20000041d:0x2:0x0] hash] 00000020:00200000:1.0:1788572496.132088:0:109627:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 9000) 0 0 00000080:00200000:1.0:1788572496.132102:0:109627:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572496.132114:0:109627:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:1.0:1788572496.132264:0:109627:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 9000) 9000, 9000 00020000:00200000:1.0:1788572496.132269:0:109627:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572496.132284:0:109627:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 2] 00000008:00010000:1.0:1788572496.132297:0:109627:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572496.132302:0:109627:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4cb0, osc lock ffff92fc47ff44c0, flags 0x0 00010000:00010000:1.0:1788572496.132323:0:109627:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572496.132338:0:109627:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 12287) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572496.132351:0:109627:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572496.132360:0:109627:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 6/0,2 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572496.132368:0:109627:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572496.132391:0:109627:0:(vvp_io.c:1292:vvp_io_write_start()) f59b.sanity-sec: write [0, 9000) append 0 append lockpos 0 00000080:00200000:1.0:1788572496.132403:0:109627:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:1.0:1788572496.132435:0:109627:0:(rw26.c:871:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:1.0:1788572496.132443:0:109627:0:(rw26.c:929:ll_write_end()) page@00000000a34ce899[3 00000000c7ebd14a 2 1 00000000373b4aff 0 0 0] 00000080:00200000:1.0:1788572496.132447:0:109627:0:(rw26.c:929:ll_write_end()) vmpage @000000007555024f 17ffffc0001001 3:0 ffff92fc47cfe000 0 lru 00000080:00200000:1.0:1788572496.132454:0:109627:0:(rw26.c:929:ll_write_end()) osc-page@0000000096b9fbf8 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000f1a46fea 000000009dd2e635 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788572496.132456:0:109627:0:(rw26.c:929:ll_write_end()) end page@00000000a34ce899 00000080:00200000:1.0:1788572496.132457:0:109627:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:1.0:1788572496.132460:0:109627:0:(rw26.c:676:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:1.0:1788572496.132490:0:109627:0:(rw26.c:871:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:1.0:1788572496.132495:0:109627:0:(rw26.c:929:ll_write_end()) page@00000000125f9352[3 00000000c7ebd14a 2 1 00000000373b4aff 0 0 1] 00000080:00200000:1.0:1788572496.132497:0:109627:0:(rw26.c:929:ll_write_end()) vmpage @000000009fd7b99c 17ffffc0001021 2:0 ffff92fc47cfea80 1 lru 00000080:00200000:1.0:1788572496.132502:0:109627:0:(rw26.c:929:ll_write_end()) osc-page@00000000b5fb23d4 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000f1a46fea 000000009dd2e635 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788572496.132504:0:109627:0:(rw26.c:929:ll_write_end()) end page@00000000125f9352 00000080:00200000:1.0:1788572496.132505:0:109627:0:(rw26.c:929:ll_write_end()) queued cl_page: 2. 00000080:00200000:1.0:1788572496.132506:0:109627:0:(rw26.c:676:ll_write_begin()) Writing 2 of 0 to 808 bytes 00000080:00200000:1.0:1788572496.132521:0:109627:0:(rw26.c:871:ll_write_end()) pos 8192, len 808, copied 808 00000080:00200000:1.0:1788572496.132525:0:109627:0:(rw26.c:929:ll_write_end()) page@000000006daf59d2[3 00000000c7ebd14a 2 1 00000000373b4aff 0 0 2] 00000080:00200000:1.0:1788572496.132528:0:109627:0:(rw26.c:929:ll_write_end()) vmpage @00000000e4a6ce9c 17ffffc0001001 3:0 ffff92fc47cfe1c0 2 lru 00000080:00200000:1.0:1788572496.132709:0:109627:0:(rw26.c:929:ll_write_end()) osc-page@00000000460e2240 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000f1a46fea 000000009dd2e635 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788572496.132712:0:109627:0:(rw26.c:929:ll_write_end()) end page@000000006daf59d2 00000080:00200000:1.0:1788572496.132714:0:109627:0:(rw26.c:929:ll_write_end()) queued cl_page: 3. 00000080:00200000:1.0:1788572496.132834:0:109627:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 3, from 0, to 808 prio 0 00000008:00010000:1.0:1788572496.132855:0:109627:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788572496.132873:0:109627:0:(osc_cache.c:2621:osc_queue_async_io()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788572496.132953:0:109627:0:(osc_cache.c:2621:osc_queue_async_io()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572496.132967:0:109627:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff92fc665db3f8, count 3, dirtied 3 00000080:00200000:1.0:1788572496.132973:0:109627:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 3 pages 9000 bytes, tot: 9000 00000080:00200000:1.0:1788572496.132976:0:109627:0:(vvp_io.c:1358:vvp_io_write_start()) f59b.sanity-sec: write bytes 0, result: 9000 00000020:00010000:1.0:1788572496.132989:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c308 (ffff92fc58255ab8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572496.132992:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c308 (ffff92fc58255ab8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572496.132995:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4cb0 (ffff92fc58255368/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572496.133000:0:109627:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572496.133010:0:109627:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:1.0:1788572496.133023:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c308 (ffff92fc58255ab8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572496.133026:0:109627:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4cb0 (ffff92fc58255ab8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572496.133034:0:109627:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000080:00200000:1.0:1788572496.133039:0:109627:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:1.0:1788572496.133044:0:109627:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572496.133048:0:109627:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 2 io complete with rc: 0, result: 9000, restart: 0 00000080:00200000:1.0:1788572496.133054:0:109627:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 0 00000080:00200000:1.0:1788572496.133058:0:109627:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 9000 00000080:00200008:1.0:1788572496.133070:0:109627:0:(file.c:5384:ll_fsync()) START file: name=f59b.sanity-sec, fid=[0x20000041d:0x2:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/1 00000080:00200000:1.0:1788572496.133078:0:109627:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572496.133083:0:109627:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1788572496.133215:0:109627:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00010000:1.0:1788572496.133228:0:109627:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572496.133269:0:109627:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:1.0:1788572496.133405:0:109627:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc6728b800 - object 0x280000401:551 offset 0<>12288 00000008:00010000:1.0:1788572496.144142:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:1.0:1788572496.144211:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000044b6b15 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00100000:1.0:1788572496.144242:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2393:1875449863824768:192.168.202.132@tcp:4:dd.0 00000100:00100000:1.0:1788572496.148001:0:109627:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job dd:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109627:1875449863824896:192.168.202.132@tcp:44:dd.0 00000080:00200000:1.0:1788572496.148036:0:109627:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572496.148106:0:109627:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788572496.148214:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572496.148229:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2393:1875449863825024:192.168.202.132@tcp:16:dd.0 00000100:00100000:1.0:1788572496.158957:0:2393:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc492a7800: committing for last_committed 4294968529 gen 1 00000100:00100000:1.0:1788572496.159031:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2393:1875449863825024:192.168.202.132@tcp:16:dd.0 00000100:00100000:1.0:1788572496.162992:0:109627:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job dd:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109627:1875449863825280:192.168.202.132@tcp:35:dd.0 00000002:00100000:1.0:1788572496.163031:0:109627:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67049880 x1875449863824000/t4294983658(4294983658) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572512 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572496.195699:0:109628:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863825536:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572496.195760:0:109628:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195775:0:109628:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195783:0:109628:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195791:0:109628:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195800:0:109628:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195807:0:109628:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572496.195811:0:109628:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195817:0:109628:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195823:0:109628:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195830:0:109628:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572496.195837:0:109628:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.195843:0:109628:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc2400/0x58a604dcef352280 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572496.195867:0:109628:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc6704a300 x1875449863825536/t0(0) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572512 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572496.195880:0:109628:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572496.196011:0:109628:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8),name=/ 00000002:00010000:2.0:1788572496.196031:0:109628:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572496.196046:0:109628:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572496.196058:0:109628:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572496.196066:0:109628:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572496.196073:0:109628:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00010000:2.0:1788572496.196076:0:109628:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) for lock 0x58a604dcef352248 00000080:00010000:2.0:1788572496.196081:0:109628:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352248 from it ffff9e03822f3bf8 00010000:00010000:2.0:1788572496.196085:0:109628:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572496.196094:0:109628:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572496.196109:0:109628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572496.196115:0:109628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:2.0:1788572496.196122:0:109628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e63c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572496.196130:0:109628:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e63c8), flags 304000 00000002:00010000:2.0:1788572496.196144:0:109628:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:2.0:1788572496.196187:0:109628:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.196195:0:109628:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572496.196206:0:109628:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc492a7800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572496.196213:0:109628:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863825792/t0(0) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572496.196222:0:109628:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572496.196240:0:109628:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863825792:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572496.196409:0:109628:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.201498:0:109628:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863825792:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572496.201526:0:109628:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201540:0:109628:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201547:0:109628:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201555:0:109628:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201573:0:109628:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201585:0:109628:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366600 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572496.201590:0:109628:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201597:0:109628:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201605:0:109628:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201612:0:109628:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572496.201620:0:109628:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572496.201626:0:109628:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70366600/0x58a604dcef352287 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109628 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572496.201646:0:109628:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617df80 x1875449863825792/t4294983662(4294983662) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572496.201658:0:109628:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617df80 x1875449863825792/t4294983662(4294983662) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572496.201672:0:109628:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572496.201692:0:109628:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572496.201716:0:109628:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617df80 x1875449863825792/t4294983662(4294983662) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572512 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572496.201829:0:109628:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e63c8) cmd=c0086915 arg=1833620 00000100:00100000:2.0:1788572496.201901:0:109628:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863825920:192.168.202.132@tcp:34:lfs.0 00000100:00100000:2.0:1788572496.201971:0:109628:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.207061:0:109628:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863825920:192.168.202.132@tcp:34:lfs.0 00000080:00200008:2.0:1788572496.207251:0:109628:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e63c8), flags 304000 00000002:00100000:2.0:1788572496.207306:0:109628:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617df80 x1875449863825792/t4294983662(4294983662) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572512 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572496.207317:0:109628:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc492a7800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572496.207323:0:109628:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863826176/t0(0) o35->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572496.207342:0:109628:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863826176:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572496.207409:0:109628:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572496.209158:0:109628:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109628:1875449863826176:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572496.209188:0:109628:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617df80 x1875449863825792/t4294983662(4294983662) o101->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572512 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572497.349296:0:109585:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572498.373479:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572499.224556:0:109651:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572499.224569:0:109651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8),name=/ 00000002:00010000:1.0:1788572499.224593:0:109651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572499.224614:0:109651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788572499.224629:0:109651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572499.224637:0:109651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572499.224644:0:109651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00010000:1.0:1788572499.224647:0:109651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) for lock 0x58a604dcef352248 00000080:00010000:1.0:1788572499.224655:0:109651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352248 from it ffff9e038236bcb8 00010000:00010000:1.0:1788572499.224659:0:109651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788572499.224668:0:109651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200008:1.0:1788572499.224835:0:109651:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572499.224840:0:109651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8),name=/ 00000002:00010000:1.0:1788572499.224849:0:109651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572499.224856:0:109651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572499.224866:0:109651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788572499.224878:0:109651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572499.224882:0:109651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00010000:1.0:1788572499.224884:0:109651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d80c8) for lock 0x58a604dcef352248 00000080:00010000:1.0:1788572499.224889:0:109651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352248 from it ffff9e038236bcb8 00010000:00010000:1.0:1788572499.224892:0:109651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572499.224899:0:109651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572499.245270:0:109651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d80c8) 00000080:00200000:1.0:1788572499.245312:0:109651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e63c8) 00000080:00200000:1.0:1788572499.245363:0:109651:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572499.245379:0:109651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572499.245400:0:109651:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572499.245460:0:109651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665db248) 00000008:00010000:1.0:1788572499.245523:0:109651:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 9000 blocks 24 [cma]time 1788572496 1788572496 0 ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 1/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572499.245584:0:109651:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc492a7800 (sb ffff92fc492a7800) 00010000:00010000:1.0:1788572499.245632:0:109651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245643:0:109651:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245657:0:109651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245665:0:109651:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245690:0:109651:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572499.245697:0:109651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245705:0:109651:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245712:0:109651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245736:0:109651:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245749:0:109651:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1788572499.245760:0:109584:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572499.245766:0:109584:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245773:0:109584:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3200 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572499.245777:0:109584:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245792:0:109584:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572499.245800:0:109584:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572499.245806:0:109584:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572499.245809:0:109584:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:1.0:1788572499.245841:0:109584:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245847:0:109584:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572499.245867:0:109584:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245875:0:109584:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3200/0x58a604dcef3521d8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fc1 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245888:0:109584:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572499.245892:0:109584:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245897:0:109584:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366800 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572499.245900:0:109584:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245907:0:109584:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572499.245914:0:109584:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572499.245919:0:109584:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572499.245921:0:109584:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:1.0:1788572499.245923:0:109584:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572499.245935:0:109584:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245941:0:109584:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572499.245946:0:109584:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245953:0:109584:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366800/0x58a604dcef352209 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fc8 expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.245965:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572499.245967:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572499.245968:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572499.245986:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:32a8c319-1bb6-4e28-859b-b22c6c45e37e:2393:1875449863826304:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572499.246081:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572499.246088:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:32a8c319-1bb6-4e28-859b-b22c6c45e37e:2393:1875449863826432:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572499.246241:0:109651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246249:0:109651:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246257:0:109651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246269:0:109651:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246280:0:109651:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:01000000:1.0:1788572499.246284:0:109651:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:1.0:1788572499.246287:0:109651:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:1.0:1788572499.246304:0:109584:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572499.246309:0:109584:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246315:0:109584:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366e00 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572499.246318:0:109584:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246325:0:109584:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572499.246332:0:109584:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572499.246337:0:109584:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572499.246339:0:109584:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:1.0:1788572499.246341:0:109584:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:1.0:1788572499.246354:0:109584:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246361:0:109584:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572499.246384:0:109584:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246394:0:109584:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366e00/0x58a604dcef352210 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fcf expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.246405:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572499.246407:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572499.246408:0:109584:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572499.246420:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:32a8c319-1bb6-4e28-859b-b22c6c45e37e:2393:1875449863826560:192.168.202.132@tcp:103:ldlm_bl.0 00020000:01000000:1.0:1788572499.247044:0:109651:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc46153000 active: 1 00020000:01000000:1.0:1788572499.247051:0:109651:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc46155000 active: 1 00020000:01000000:1.0:1788572499.247054:0:109651:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:1.0:1788572499.247060:0:109651:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc492a7800 00020000:01000000:1.0:1788572499.247062:0:109651:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc492a7800: disconnecting target lustre-OST0001-osc-ffff92fc492a7800 00000100:00080000:1.0:1788572499.247073:0:109651:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140->lustre-OST0001_UUID 00000100:00080000:1.0:1788572499.247170:0:109651:0:(import.c:65:import_set_state_nolock()) ffff92fc46153800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572499.247201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2392:1875449863826688:192.168.202.132@tcp:9:umount.0 00000100:00080000:2.0:1788572499.266353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46153800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572499.266398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:2392:1875449863826688:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572499.266679:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572499.266683:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc492a7800: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572499.266689:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.266869:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35223a 00000020:00000080:1.0:1788572499.266888:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc46155000/LOV_OSC_UUID 00000020:01000000:1.0:1788572499.266934:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc492a7800 (flags='') 00000020:00000080:1.0:1788572499.266939:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.266948:0:109651:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc492a7800: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572499.266952:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc70033380) has no exports 00000008:01000000:1.0:1788572499.267376:0:109651:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc492a7800: client import never connected 00000100:00080000:1.0:1788572499.267380:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572499.267383:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc492a7800: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572499.267386:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.267575:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.267582:0:109651:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00020000:01000000:1.0:1788572499.267586:0:109651:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc492a7800 00020000:01000000:1.0:1788572499.267588:0:109651:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc492a7800: disconnecting target lustre-OST0000-osc-ffff92fc492a7800 00000100:00080000:1.0:1788572499.267597:0:109651:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140->lustre-OST0000_UUID 00010000:00010000:1.0:1788572499.267626:0:109651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572499.267676:0:109651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x227:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572499.267685:0:109651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572499.267698:0:109651:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705556c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.267707:0:109651:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job umount:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109651:1875449863826816:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572499.267715:0:109651:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572499.267765:0:109651:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572499.267769:0:109651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc492a7800 lock: 000000000268bc32/0x58a604dcef352279 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x227:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217920b6 expref: -99 pid: 109627 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00080000:1.0:1788572499.267855:0:109651:0:(import.c:65:import_set_state_nolock()) ffff92fc46150000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572499.275455:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572499.275460:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc492a7800: committing for last_committed 4294968529 gen 2 00000100:00100000:1.0:1788572499.275464:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.277570:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352233 00000020:00000080:1.0:1788572499.277704:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc46153000/LOV_OSC_UUID 00000020:01000000:1.0:1788572499.277790:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc492a7800 (flags='') 00000020:00000080:1.0:1788572499.277798:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.277809:0:109651:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc492a7800: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572499.277814:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc700319c0) has no exports 00000008:01000000:1.0:1788572499.279676:0:109651:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc492a7800: client import never connected 00000100:00080000:1.0:1788572499.279684:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572499.279689:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc492a7800: committing for last_committed 4294968529 gen 3 00000100:00100000:1.0:1788572499.279698:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.279903:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.279912:0:109651:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:00000080:1.0:1788572499.279918:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35222c 00000020:00000080:1.0:1788572499.279927:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc46157000/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 00000100:00080000:1.0:1788572499.279989:0:109651:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140->lustre-MDT0000_UUID 00010000:00010000:1.0:1788572499.280100:0:109651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572499.280149:0:109651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572499.280157:0:109651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572499.280172:0:109651:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705556c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.280183:0:109651:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job umount:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109651:1875449863827072:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572499.280192:0:109651:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572499.280199:0:109651:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572499.280203:0:109651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70205200/0x58a604dcef352248 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179201c expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572499.280839:0:109651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204a00/0x58a604dcef35224f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792038 expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572499.280930:0:109651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204a00/0x58a604dcef35224f lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792038 expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572499.280938:0:109651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572499.280947:0:109651:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705556c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.280954:0:109651:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job umount:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109651:1875449863827200:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572499.280962:0:109651:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572499.280968:0:109651:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572499.280971:0:109651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70204a00/0x58a604dcef35224f lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792038 expref: -99 pid: 109626 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572499.281004:0:109651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc3800/0x58a604dcef352272 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217920a8 expref: -99 pid: 109627 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572499.281015:0:109651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572499.281028:0:109651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc3800/0x58a604dcef352272 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217920a8 expref: -99 pid: 109627 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572499.281036:0:109651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572499.281041:0:109651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:1.0:1788572499.281046:0:109651:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705556c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.281050:0:109651:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job umount:2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140:109651:1875449863827328:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572499.281055:0:109651:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572499.282017:0:109651:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572499.282024:0:109651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc51bc3800/0x58a604dcef352272 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217920a8 expref: -99 pid: 109627 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572499.282037:0:109651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc492a7800 lock: ffff92fc70367800/0x58a604dcef352264 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792077 expref: -99 pid: 109626 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000100:00080000:1.0:1788572499.284469:0:109651:0:(import.c:65:import_set_state_nolock()) ffff92fc69d47000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572499.290619:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572499.290623:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc492a7800: committing for last_committed 4294983657 gen 2 00000100:00100000:1.0:1788572499.290634:0:109651:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67048e00 x1875449863824384/t4294983660(4294983660) o36->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572512 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572499.290655:0:109651:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6728a680 x1875449863827456/t4294983664(4294983664) o39->lustre-MDT0000-mdc-ffff92fc492a7800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572509 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572499.290668:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.296311:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35221e 00000020:00000080:1.0:1788572499.296332:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc4a3b6800/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 00000100:00080000:1.0:1788572499.296526:0:109651:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140->lustre-MDT0001_UUID 00000100:00080000:1.0:1788572499.303136:0:109651:0:(import.c:65:import_set_state_nolock()) ffff92fc46154800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572499.310767:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572499.310772:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc492a7800: committing for last_committed 4294976744 gen 2 00000100:00100000:1.0:1788572499.310778:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.312649:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352225 00000020:00000080:1.0:1788572499.312664:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc4a3b0800/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 00000020:00000080:1.0:1788572499.312697:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352217 00000020:00000080:1.0:1788572499.312700:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc4a3b7000/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 00000020:01000000:1.0:1788572499.312709:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc492a7800 (flags='') 00000020:00000080:1.0:1788572499.312718:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.312751:0:109651:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc492a7800: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572499.312756:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc48d10000) has no exports 00000020:00000080:1.0:1788572499.312827:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.312834:0:109651:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:00000080:1.0:1788572499.312838:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc69d44800/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 00000020:00000080:1.0:1788572499.312841:0:109651:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d44800/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 for lustre-clilov-ffff92fc492a7800 00000020:01000000:1.0:1788572499.312847:0:109651:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc492a7800 (2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:01000000:1.0:1788572499.313024:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc492a7800 (flags='') 00000020:00000080:1.0:1788572499.313028:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.313031:0:109651:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc492a7800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572499.313034:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc48d13380) has no exports 00000020:00000080:1.0:1788572499.313084:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.313087:0:109651:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:01000000:1.0:1788572499.313091:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc492a7800 (flags='') 00000020:00000080:1.0:1788572499.313093:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.313096:0:109651:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc492a7800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572499.313098:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc48d119c0) has no exports 00000020:00000080:2.0:1788572499.314184:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4a3b0800/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 for lustre-MDT0001-mdc-ffff92fc492a7800 00000020:00000080:2.0:1788572499.314201:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4a3b7000/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 for lustre-clilmv-ffff92fc492a7800 00000020:00000080:2.0:1788572499.314207:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc69d40000/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 00000020:00000080:2.0:1788572499.314210:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d40000/2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140 for lustre-clilmv-ffff92fc492a7800 00000020:01000000:2.0:1788572499.314214:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc492a7800 (2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000008:01000000:1.0:1788572499.318575:0:109651:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc492a7800: client import never connected 00000100:00080000:1.0:1788572499.320104:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572499.320108:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc492a7800: committing for last_committed 4294983657 gen 3 00000100:00100000:1.0:1788572499.320119:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.324649:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.324664:0:109651:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000020:01000000:1.0:1788572499.324680:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc492a7800 (flags='') 00000020:00000080:1.0:1788572499.324684:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.324688:0:109651:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc492a7800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572499.324693:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc70030000) has no exports 00000008:01000000:1.0:1788572499.330068:0:109651:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc492a7800: client import never connected 00000100:00080000:1.0:1788572499.330080:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572499.330086:0:109651:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc492a7800: committing for last_committed 4294976744 gen 3 00000100:00100000:1.0:1788572499.330098:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572499.344191:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.344206:0:109651:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc492a7800 (uuid 2f002d62-6a8d-4ce0-8dbd-f1c4ba2eb140) 00000080:02000400:1.0:1788572499.344213:0:109651:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572499.344216:0:109651:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1788572499.344264:0:109651:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc492a7800 00000100:00080000:1.0:1788572499.344272:0:109651:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 32a8c319-1bb6-4e28-859b-b22c6c45e37e->MGS 00010000:00010000:1.0:1788572499.344285:0:109651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef3521d1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221791fba expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572499.344298:0:109651:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef3521d1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221791fba expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572499.344305:0:109651:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572499.344308:0:109651:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572499.344311:0:109651:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572499.344356:0:109651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef3521d1 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fba expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.344364:0:109651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572499.344391:0:109651:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70555900 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.344401:0:109651:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job umount:32a8c319-1bb6-4e28-859b-b22c6c45e37e:109651:1875449863827712:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572499.344410:0:109651:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572499.344418:0:109651:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572499.344422:0:109651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef3521d1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221791fba expref: -99 pid: 109573 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572499.344448:0:109651:0:(import.c:65:import_set_state_nolock()) ffff92fc5df75000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572499.344472:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:32a8c319-1bb6-4e28-859b-b22c6c45e37e:2392:1875449863827840:192.168.202.132@tcp:251:umount.0 00000100:00080000:1.0:1788572499.353948:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5df75000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:1.0:1788572499.353960:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788572499.353963:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788572499.353965:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572499.353977:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:32a8c319-1bb6-4e28-859b-b22c6c45e37e:2392:1875449863827840:192.168.202.132@tcp:251:umount.0 00000100:00080000:1.0:1788572499.354019:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572499.354023:0:109651:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572499.354025:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572499.354027:0:109651:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572499.354038:0:109651:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3521ca 00000020:00000080:1.0:1788572499.354046:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc492a0000/32a8c319-1bb6-4e28-859b-b22c6c45e37e 00000020:01000000:1.0:1788572499.354062:0:109651:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:1.0:1788572499.354068:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572499.354075:0:109651:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572499.354079:0:109651:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc499319c0) has no exports 00000020:00000080:1.0:1788572499.359986:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a0000/32a8c319-1bb6-4e28-859b-b22c6c45e37e for MGC192.168.202.132@tcp 10000000:01000000:1.0:1788572499.361781:0:109587:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:1.0:1788572499.361936:0:109651:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:1.0:1788572499.361941:0:109651:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572499.361945:0:109651:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572499.361949:0:109651:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572499.361951:0:109651:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572499.361991:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5df75000 for MGC192.168.202.132@tcp 00000020:00000080:1.0:1788572499.362051:0:109651:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572499.362060:0:109651:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 32a8c319-1bb6-4e28-859b-b22c6c45e37e) 00000020:00000080:1.0:1788572499.362065:0:109651:0:(genops.c:885:class_export_put()) final put ffff92fc492a1000/32a8c319-1bb6-4e28-859b-b22c6c45e37e 00000020:00000080:1.0:1788572499.362069:0:109651:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a1000/32a8c319-1bb6-4e28-859b-b22c6c45e37e for MGC192.168.202.132@tcp 00000020:01000000:1.0:1788572499.362074:0:109651:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (32a8c319-1bb6-4e28-859b-b22c6c45e37e) 00010000:00010000:1.0:1788572499.362320:0:109584:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572499.363777:0:109585:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00000100:00100000:1.0:1788572499.364115:0:109575:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc48dd2a00:109575] 1 exiting: rc = 0 00000100:00100000:1.0:1788572499.365822:0:109574:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc47f04800:109574] 0 exiting: rc = 1 00000100:00100000:2.0:1788572499.366052:0:109576:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc47f04a00:109576] 0 exiting: rc = 1 00000100:00100000:2.0:1788572499.366680:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.366693:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572499.367160:0:109578:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc49c87200:109578] 0 exiting: rc = 1 00000100:00100000:1.0:1788572499.367823:0:109579:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc53dfea00:109579] 1 exiting: rc = 0 00000100:00100000:1.0:1788572499.369847:0:109580:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc53dfe600:109580] 2 exiting: rc = 0 00000100:00100000:2.0:1788572499.370149:0:109581:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc53dffc00:109581] 0 exiting: rc = 1 00000100:00100000:2.0:1788572499.371007:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371016:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371021:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371027:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371032:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371037:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371042:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371047:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371052:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371060:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371065:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371071:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371076:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371081:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371087:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371092:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371098:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371103:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371111:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371117:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371123:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371129:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371134:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371138:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371143:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371149:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371154:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371159:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371165:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371169:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371175:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371179:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371184:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371189:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371194:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371200:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371205:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371211:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371217:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371222:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371227:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371233:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371238:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371243:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371248:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371253:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371259:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371264:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371270:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371275:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371280:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371285:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371290:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371295:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371300:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371305:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371310:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371315:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371320:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371325:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371329:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371334:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371338:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371343:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371350:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371355:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371361:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371366:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371383:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371388:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371394:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371399:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371405:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371410:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371416:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371421:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371426:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371431:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371436:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371441:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371445:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371450:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371454:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371460:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371466:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371472:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371477:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371482:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371488:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371493:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371498:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371503:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371509:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371513:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371519:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371525:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371530:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371535:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371540:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371545:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371551:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371556:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371561:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371567:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371572:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371577:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371594:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371600:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371606:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371612:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371618:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371623:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371629:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371635:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371641:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371646:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371651:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371657:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371663:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371667:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371673:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371678:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371684:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371689:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371694:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371700:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371705:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572499.371710:0:109651:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:2.0:1788572499.372980:0:109651:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc492a7800 1 00000020:01000004:2.0:1788572499.372987:0:109651:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc492a3800 00000100:00100000:1.0:1788572499.573423:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:1.0:1788572499.573436:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:1.0:1788572499.574684:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572499.574690:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572499.576339:0:109664:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572499.576542:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572499.576549:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788572499.577512:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788572499.577521:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:2.0:1788572499.578073:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:2.0:1788572499.578079:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788572499.578431:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788572499.578437:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:1.0:1788572499.579190:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:1.0:1788572499.579198:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:2.0:1788572499.579658:0:109664:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:2.0:1788572499.579664:0:109664:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00000020:00000080:2.0:1788572499.586232:0:109664:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid 9e588565-c27f-4f6e-85de-718639cb4969) 00000020:00000080:2.0:1788572499.586258:0:109664:0:(genops.c:1385:class_connect()) connect: client 9e588565-c27f-4f6e-85de-718639cb4969, cookie 0x58a604dcef352295 00000100:00080000:2.0:1788572499.586274:0:109664:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572499.586288:0:109664:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc51c44000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572499.586324:0:109664:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449863827968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572499.586351:0:109664:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 9e588565-c27f-4f6e-85de-718639cb4969->MGS 00000080:00200000:2.0:1788572499.586384:0:109664:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc4990d800 (sb ffff92fc4990d800) 00000080:01000000:2.0:1788572499.586455:0:109664:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:2.0:1788572499.586458:0:109664:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:2.0:1788572499.586464:0:109664:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:2.0:1788572499.586473:0:109664:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 543b1dda-86fa-421e-9749-35cac38dec65 00000100:00100000:1.0:1788572499.586994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9e588565-c27f-4f6e-85de-718639cb4969:2392:1875449863827968:192.168.202.132@tcp:250:mount.lustre.0 10000000:01000000:2.0:1788572499.588279:0:109664:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:2.0:1788572499.588284:0:109664:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc4990d800 10000000:01000000:2.0:1788572499.588289:0:109664:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc48cc8000 10000000:01000000:2.0:1788572499.588294:0:109664:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572499.588302:0:109664:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc48cc8000 from 1 10000000:01000000:2.0:1788572499.588305:0:109664:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788572499.588348:0:109664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572499.588359:0:109664:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572499.588381:0:109664:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572499.588406:0:109664:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc4617df80 x1875449863828096/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572499.588414:0:109664:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6038d6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.599051:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572499.599070:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:1.0:1788572499.599079:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:1.0:1788572499.599085:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:1.0:1788572499.599089:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51c44000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:1.0:1788572499.599093:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:1.0:1788572499.599102:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:1.0:1788572499.599183:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000058ff01f2): req@ffff92fc4617df80 x1875449863828096/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572499.601970:0:109664:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6038d6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572499.605230:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9e588565-c27f-4f6e-85de-718639cb4969:2392:1875449863827968:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572499.613851:0:109664:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9e588565-c27f-4f6e-85de-718639cb4969:109664:1875449863828096:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572499.613954:0:109664:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217920fc expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572499.613971:0:109664:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217920fc expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572499.614053:0:109664:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9e588565-c27f-4f6e-85de-718639cb4969:109664:1875449863828224:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572499.624256:0:109664:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9e588565-c27f-4f6e-85de-718639cb4969:109664:1875449863828224:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:1.0:1788572499.624300:0:109664:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788572499.624324:0:109664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217920fc expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.624337:0:109664:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef35229c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217920fc expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572499.624347:0:109664:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc4990d800 10000000:01000000:1.0:1788572499.624355:0:109664:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:1.0:1788572499.624359:0:109664:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc4990d800 10000000:01000000:1.0:1788572499.624362:0:109664:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788572499.624376:0:109664:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc4990d800 10000000:01000000:1.0:1788572499.624378:0:109664:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:1.0:1788572499.624382:0:109664:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc4990d800 from 1 10000000:01000000:1.0:1788572499.624385:0:109664:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572499.624423:0:109664:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.624431:0:109664:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.624443:0:109664:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572499.624464:0:109664:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9e588565-c27f-4f6e-85de-718639cb4969:109664:1875449863828352:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788572499.624545:0:109664:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58de16c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572499.663081:0:109682:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4991e3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572499.674093:0:109682:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9e588565-c27f-4f6e-85de-718639cb4969:109682:1875449863828864:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572499.674829:0:109682:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572499.674841:0:109682:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572499.674845:0:109682:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572499.674850:0:109682:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572499.674910:0:109682:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9e588565-c27f-4f6e-85de-718639cb4969:109682:1875449863828992:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572499.675014:0:109682:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384eb40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572499.685541:0:109682:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9e588565-c27f-4f6e-85de-718639cb4969:109682:1875449863828992:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572499.685582:0:109682:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572499.685591:0:109682:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572499.685594:0:109682:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:2.0:1788572499.685598:0:109682:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000100:00100000:1.0:1788572499.693994:0:109664:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9e588565-c27f-4f6e-85de-718639cb4969:109664:1875449863829120:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572499.694026:0:109664:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.694045:0:109664:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:01000000:1.0:1788572499.728688:0:109664:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:1.0:1788572499.728718:0:109664:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788572499.728763:0:109664:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572499.728774:0:109664:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788572499.728798:0:109664:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1788572499.728823:0:109664:0:(genops.c:1385:class_connect()) connect: client 543b1dda-86fa-421e-9749-35cac38dec65, cookie 0x58a604dcef3522e2 00800000:01000000:1.0:1788572499.728847:0:109664:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 543b1dda-86fa-421e-9749-35cac38dec65 to lustre-clilmv-ffff92fc4990d800 00800000:01000000:1.0:1788572499.728855:0:109664:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc4990d800(543b1dda-86fa-421e-9749-35cac38dec65) - lustre-MDT0000_UUID, 543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:1.0:1788572499.729021:0:109664:0:(genops.c:1385:class_connect()) connect: client 543b1dda-86fa-421e-9749-35cac38dec65, cookie 0x58a604dcef3522e9 00000100:00080000:1.0:1788572499.729032:0:109664:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc4990d800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572499.729042:0:109664:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc4990d800: import ffff92fc4a3b7000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572499.729054:0:109664:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572499.729082:0:109664:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449863830016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572499.729103:0:109664:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 543b1dda-86fa-421e-9749-35cac38dec65->lustre-MDT0000_UUID 00800000:01000000:1.0:1788572499.729256:0:109664:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc4990d800(543b1dda-86fa-421e-9749-35cac38dec65) successfully (5) 00800000:00000004:1.0:1788572499.729260:0:109664:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc4990d800: age 3255, max_age 0 00800000:01000000:1.0:1788572499.729968:0:109664:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc4990d800(543b1dda-86fa-421e-9749-35cac38dec65) - lustre-MDT0001_UUID, 543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:1.0:1788572499.729979:0:109664:0:(genops.c:1385:class_connect()) connect: client 543b1dda-86fa-421e-9749-35cac38dec65, cookie 0x58a604dcef3522f0 00000100:00080000:1.0:1788572499.729987:0:109664:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc4990d800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572499.729994:0:109664:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc4990d800: import ffff92fc58388000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572499.730003:0:109664:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572499.730019:0:109664:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449863830272/t0(0) o38->lustre-MDT0001-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572499.730028:0:109664:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 543b1dda-86fa-421e-9749-35cac38dec65->lustre-MDT0001_UUID 00800000:01000000:1.0:1788572499.730078:0:109664:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc4990d800(543b1dda-86fa-421e-9749-35cac38dec65) successfully (5) 00800000:00000004:1.0:1788572499.730081:0:109664:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc4990d800: age 3255, max_age 0 00000080:00000004:1.0:1788572499.736212:0:109664:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc4990d800: age 3255, max_age 4254 00800000:00000004:1.0:1788572499.736224:0:109664:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc4990d800: age 3255, max_age 4254 00000100:00080000:1.0:1788572499.736261:0:109664:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6728a680 x1875449863830528/t0(0) o41->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572499.736270:0:109664:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4245af00 going to sleep for 0 seconds 00000100:00100000:1.0:1788572499.736303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830016:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572499.736437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830272:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572499.748338:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4990d800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572499.748357:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc4990d800: connect to target with instance 1 00000100:00080000:2.0:1788572499.748377:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc4990d800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572499.748390:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1788572499.748394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4a3b7000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:1.0:1788572499.750329:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc6728b480 x1875449863830144/t0(0) o41->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572499.750404:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000001cd92299): req@ffff92fc6728a680 x1875449863830528/t0(0) o41->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572499.750698:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830016:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572499.750755:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4990d800: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572499.750765:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc4990d800: connect to target with instance 1 00000100:00080000:1.0:1788572499.750776:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc4990d800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:1.0:1788572499.750789:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:1.0:1788572499.750792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58388000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788572499.751610:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc67288e00 x1875449863830400/t0(0) o41->lustre-MDT0001-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572499.751626:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830272:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572499.756679:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:543b1dda-86fa-421e-9749-35cac38dec65:0:1875449863830400:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572499.756696:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:543b1dda-86fa-421e-9749-35cac38dec65:0:1875449863830144:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572499.761786:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830784:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572499.768052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830912:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572499.771236:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4990d800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572499.771250:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc4990d800: connect to target with instance 1 00000100:00080000:2.0:1788572499.771259:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc4990d800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572499.771271:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572499.771274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5838a000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572499.771280:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1788572499.771284:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572499.771288:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572499.771304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830784:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572499.774395:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4990d800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572499.774414:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc4990d800: connect to target with instance 1 00000100:00080000:2.0:1788572499.774425:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc4990d800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572499.774439:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788572499.774443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5838d000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572499.774451:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:2.0:1788572499.774457:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572499.774462:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572499.774475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863830912:192.168.202.132@tcp:8:mount.lustre.0 00010000:00010000:1.0:1788572500.045916:0:109719:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:1.0:1788572500.045968:0:109719:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc4990d800 00010000:00010000:1.0:1788572500.045995:0:109719:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc4990d800 00010000:00010000:1.0:1788572500.046022:0:109719:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc4990d800 00010000:00010000:1.0:1788572500.046050:0:109719:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc4990d800 00010000:00010000:1.0:1788572500.613319:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788572500.702622:0:109726:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8),name=/ 00000002:00010000:1.0:1788572500.702654:0:109726:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572500.702711:0:109726:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109726 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.702755:0:109726:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109726 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.702767:0:109726:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109726 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572500.702792:0:109726:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863831296:192.168.202.132@tcp:101:find.0 00000100:00100000:1.0:1788572500.702859:0:109726:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4245ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572500.706351:0:109726:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863831296:192.168.202.132@tcp:101:find.0 00010000:00010000:1.0:1788572500.706390:0:109726:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706404:0:109726:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706416:0:109726:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706432:0:109726:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706444:0:109726:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706452:0:109726:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706460:0:109726:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572500.706472:0:109726:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b480 x1875449863831296/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572516 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'find.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572500.706486:0:109726:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706499:0:109726:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc671e4c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572500.706505:0:109726:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572500.706533:0:109726:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572500.706538:0:109726:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef35230c 00000080:00010000:1.0:1788572500.706543:0:109726:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:1.0:1788572500.706546:0:109726:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35230c from it ffff9e0382ed3b58 00010000:00010000:1.0:1788572500.706549:0:109726:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706557:0:109726:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572500.706583:0:109726:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572500.706605:0:109726:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665de3c8), flags=11 00000080:00200000:1.0:1788572500.706610:0:109726:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665de3c8), intent=getattr 00000002:00010000:1.0:1788572500.706632:0:109726:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572500.706673:0:109726:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109726 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706680:0:109726:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109726 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.706688:0:109726:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109726 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572500.706706:0:109726:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863831424:192.168.202.132@tcp:101:find.0 00000100:00100000:1.0:1788572500.706888:0:109726:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4245ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572500.713491:0:109726:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863831424:192.168.202.132@tcp:101:find.0 00010000:00010000:1.0:1788572500.713522:0:109726:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713550:0:109726:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713558:0:109726:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713570:0:109726:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713583:0:109726:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713597:0:109726:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713605:0:109726:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713612:0:109726:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572500.713624:0:109726:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b480 x1875449863831424/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572516 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'find.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572500.713637:0:109726:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713650:0:109726:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc671e4c00 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:1.0:1788572500.713657:0:109726:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572500.713690:0:109726:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d21c8) 00000080:00200000:1.0:1788572500.713699:0:109726:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572500.713717:0:109726:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d21c8) for lock 0x58a604dcef352313 00000080:00010000:1.0:1788572500.713827:0:109726:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d21c8) 00000080:00010000:1.0:1788572500.713832:0:109726:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352313 from it ffff9e0382ed3b20 00010000:00010000:1.0:1788572500.713835:0:109726:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572500.713845:0:109726:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572500.743299:0:109726:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863832192:192.168.202.132@tcp:37:find.0 00010000:00010000:2.0:1788572500.744241:0:109726:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572500.744257:0:109726:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f4600/0x58a604dcef352313 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792157 expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572500.744331:0:109726:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d21c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:2.0:1788572500.744493:0:109726:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d21c8), flags 704000 00000002:00100000:2.0:1788572500.744557:0:109726:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67339500 x1875449863831808/t4294983665(4294983665) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572516 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'find.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572500.744569:0:109726:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572500.744574:0:109726:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ce00 x1875449863832448/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572500.744596:0:109726:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863832448:192.168.202.132@tcp:35:find.0 00000100:00100000:2.0:1788572500.744658:0:109726:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384ea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572500.747066:0:109726:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job find:543b1dda-86fa-421e-9749-35cac38dec65:109726:1875449863832448:192.168.202.132@tcp:35:find.0 00000002:00100000:2.0:1788572500.747097:0:109726:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67339500 x1875449863831808/t4294983665(4294983665) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572516 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'find.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572500.932074:0:109727:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8),name=/ 00000002:00010000:2.0:1788572500.932226:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572500.932251:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572500.932268:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572500.932282:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572500.932291:0:109727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:2.0:1788572500.932294:0:109727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef35230c 00000080:00010000:2.0:1788572500.932301:0:109727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35230c from it ffff9e0382edbbf8 00010000:00010000:2.0:1788572500.932305:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572500.932313:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572500.932327:0:109727:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572500.932335:0:109727:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:2.0:1788572500.932341:0:109727:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d21c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572500.932354:0:109727:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d21c8), file ffff92fc47380180, open_flags 4d902, mode 0 opened 0 00000080:00200000:2.0:1788572500.932372:0:109727:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d21c8), intent=open 00000002:00010000:2.0:1788572500.932429:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 01154403 00010000:00010000:2.0:1788572500.932487:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.932496:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572500.932509:0:109727:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572500.932522:0:109727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ce00 x1875449863832704/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572500.932532:0:109727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572500.932555:0:109727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863832704:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572500.933017:0:109727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384ea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572500.958357:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863832704:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572500.958398:0:109727:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572500.979330:0:109665:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881859584 00000100:00100000:1.0:1788572500.979339:0:109665:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881859584 00000100:00100000:1.0:1788572500.979409:0:109665:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:1.0:1788572500.979420:0:109665:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67048000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:543b1dda-86fa-421e-9749-35cac38dec65+5:44285:x1875449881859584:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572500.979438:0:109665:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70205800/0x58a604dcef352328 lrc: 2/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792196 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572500.979521:0:109665:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572500.979528:0:109665:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67048000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:543b1dda-86fa-421e-9749-35cac38dec65+5:44285:x1875449881859584:12345-192.168.202.132@tcp:104: Request processed in 110us (402us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572500.979536:0:109665:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:2.0:1788572500.989423:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863833344:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572500.989448:0:109727:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989463:0:109727:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989470:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989477:0:109727:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989487:0:109727:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989492:0:109727:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572500.989497:0:109727:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989505:0:109727:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989513:0:109727:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989520:0:109727:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572500.989527:0:109727:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.989534:0:109727:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204200/0x58a604dcef35232f lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572500.989564:0:109727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff92fc6733a680 x1875449863833344/t4294983669(4294983669) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572516 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572500.989577:0:109727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6733a680 x1875449863833344/t4294983669(4294983669) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572516 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572500.989594:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572500.989678:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041e:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572500.989685:0:109727:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041e:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572500.989766:0:109727:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041e:0x1:0x0], updating i_size 0 00000002:00100000:2.0:1788572500.989784:0:109727:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6733a680 x1875449863833344/t4294983669(4294983669) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572516 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572500.989821:0:109727:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6733a680 x1875449863833344/t4294983669(4294983669) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572516 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572500.989830:0:109727:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572500.989835:0:109727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c700 x1875449863833728/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572500.989855:0:109727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863833728:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572500.989924:0:109727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572500.993979:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863833728:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572500.994026:0:109727:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6733a680 x1875449863833344/t4294983669(4294983669) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572516 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572500.994059:0:109727:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000041e:0x1:0x0](ffff92fc665da1c8) 00000002:00010000:2.0:1788572500.994070:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041e:0x1:0x0]) in obj [0x20000041e:0x1:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572500.994125:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: --/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.994137:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: --/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572500.994147:0:109727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: --/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572500.994165:0:109727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863833856:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572500.994233:0:109727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.010166:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863833856:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572501.010201:0:109727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 4/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.010217:0:109727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 4/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572501.010232:0:109727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff92fc4617d500 x1875449863833856/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572516 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572501.010243:0:109727:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572501.010257:0:109727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.010270:0:109727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc671e4c00 type 13 mode 16 res 8589935646/1 (0 0) 00000080:00200000:2.0:1788572501.010277:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000041e:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572501.010296:0:109727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041e:0x1:0x0](ffff92fc665da1c8) for lock 0x58a604dcef352336 00000080:00010000:2.0:1788572501.010304:0:109727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352336 from it ffff9e0382edbc38 00010000:00010000:2.0:1788572501.010309:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.010319:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 2/0,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.010343:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 2/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.010355:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 2/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572501.010376:0:109727:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000041e:0x1:0x0](00000000c2c9ec34) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572501.010413:0:109727:0:(file.c:6843:ll_layout_conf()) [0x20000041e:0x1:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788572501.010419:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 3/1,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.010426:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 2/0,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572501.010448:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041e:0x1:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572501.010460:0:109727:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572501.010539:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572501.010556:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863833984:192.168.202.132@tcp:2:lfs.0 00000100:00100000:2.0:1788572501.031391:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863833984:192.168.202.132@tcp:2:lfs.0 00000002:00010000:2.0:1788572501.038250:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572501.038527:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.038543:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572501.038552:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572501.038561:0:109727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665daa08) 00000080:00010000:2.0:1788572501.038565:0:109727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665daa08) for lock 0x58a604dcef352321 00000080:00010000:2.0:1788572501.038573:0:109727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352321 from it ffff9e0382edbcd8 00010000:00010000:2.0:1788572501.038577:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.038585:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572501.038612:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572501.038623:0:109727:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572501.038675:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572501.038680:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550db0, osc lock ffff92fc58371300, flags 0x400001002 00010000:00010000:2.0:1788572501.039536:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.039550:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572501.039802:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572501.056577:0:2395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.056605:0:2395:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.056620:0:2395:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.056640:0:2395:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572501.056656:0:2395:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572501.056668:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572501.056813:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.056828:0:2395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.056849:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.056865:0:2395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572501.056885:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863834112:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572501.076648:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:543b1dda-86fa-421e-9749-35cac38dec65:2394:1875449863834368:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572501.083284:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.083321:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.083337:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.083355:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572501.083380:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572501.083392:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788572501.083412:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.083421:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.083429:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.083447:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 00000000bdce130e/0x58a604dcef35234b lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217921d5 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572501.083473:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:543b1dda-86fa-421e-9749-35cac38dec65:2394:1875449863834368:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572501.087056:0:109665:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881859712 00000100:00100000:1.0:1788572501.087076:0:109665:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881859712 00000100:00100000:1.0:1788572501.087142:0:109665:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 1 00000100:00100000:1.0:1788572501.087153:0:109665:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67048a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8446:x1875449881859712:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572501.087174:0:109665:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572501.087271:0:109665:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572501.087280:0:109665:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67048a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8446:x1875449881859712:12345-192.168.202.132@tcp:104: Request processed in 128us (367us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572501.087287:0:109665:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 1 00010000:00010000:2.0:1788572501.087891:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572501.087900:0:109676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572501.087910:0:109676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367800 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:2.0:1788572501.087918:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572501.087979:0:109676:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572501.087992:0:109676:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572501.088038:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 2/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572501.088051:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572501.088085:0:109676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 1/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572501.088094:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35233d lrc: 0/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a2217921c0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572501.089555:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.089559:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.089561:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572501.090878:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572501.090911:0:109665:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881859840 00000100:00100000:1.0:1788572501.090919:0:109665:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881859840 00000100:00100000:1.0:1788572501.090973:0:109665:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 2 00000100:00100000:1.0:1788572501.090983:0:109665:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6728b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6515:x1875449881859840:12345-192.168.202.132@tcp:105: 00010000:00010000:1.0:1788572501.090996:0:109665:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,1 mode: --/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x20000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.091016:0:109665:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,1 mode: --/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x20000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.091030:0:109665:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,1 mode: --/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x10000000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.091040:0:109665:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x10080000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.091050:0:109665:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 6/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x10080000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.091075:0:109665:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 6/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x80000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572501.091100:0:109665:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff92fc51bc2200) 00000100:00100000:1.0:1788572501.091351:0:109665:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6728b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6515:x1875449881859840:12345-192.168.202.132@tcp:105: Request processed in 368us (476us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572501.091359:0:109665:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 2 00010000:00010000:2.0:1788572501.091550:0:109727:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 4/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x20000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.091562:0:109727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 4/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x20000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.091579:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x10000000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572501.091591:0:109727:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 4/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x10020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.091604:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.091617:0:109727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 4/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1788572501.091656:0:109727:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665daa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572501.091665:0:109727:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665daa08),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572501.091740:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572501.091756:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572501.091769:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572501.091778:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572501.091785:0:109727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665daa08) 00000080:00010000:2.0:1788572501.091789:0:109727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665daa08) for lock 0x58a604dcef352321 00000080:00010000:2.0:1788572501.091795:0:109727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352321 from it ffff9e0382edbcd8 00010000:00010000:2.0:1788572501.091798:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572501.091806:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572501.091830:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572501.091838:0:109727:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572501.091854:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572501.091857:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550d30, osc lock ffff92fc58370000, flags 0x400001002 00010000:00010000:2.0:1788572501.091868:0:109727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245c00 type 11 mode 6 res 10737419265/551 (0 18446744073709551615) 00010000:00010000:2.0:1788572501.091897:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.091906:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572501.092371:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788572501.100874:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00010000:00010000:2.0:1788572501.107333:0:2395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.107359:0:2395:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.107384:0:2395:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.107392:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.107399:0:2395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572501.107414:0:2395:0:(osc_lock.c:187:osc_lock_lvb_update()) ### no dlm lock: acquired size 9000, setting rss=9000; leaving kms=9000, end=18446744073709551615 00000020:00200000:2.0:1788572501.107421:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572501.107449:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572501.107822:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:543b1dda-86fa-421e-9749-35cac38dec65:2393:1875449863834880:192.168.202.132@tcp:1:lfs.0 00010000:00010000:2.0:1788572501.108517:0:2395:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.108529:0:2395:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.108536:0:2395:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366e00 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:2.0:1788572501.108542:0:2395:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.108582:0:2395:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572501.108590:0:2395:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.108599:0:2395:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352359 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572501.108615:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863834752:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572501.122703:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:543b1dda-86fa-421e-9749-35cac38dec65:2393:1875449863834880:192.168.202.132@tcp:1:lfs.0 00000080:00200000:2.0:1788572501.123701:0:109727:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665daa08), inode mode 81a4 mask 4 00000080:00200000:2.0:1788572501.123715:0:109727:0:(xattr.c:744:ll_xattr_get()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665daa08), xattr lov 00000080:00200008:2.0:1788572501.123782:0:109727:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665daa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572501.123790:0:109727:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665daa08),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572501.123852:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572501.123875:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572501.123891:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572501.123899:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572501.123906:0:109727:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665daa08) 00000080:00010000:2.0:1788572501.123910:0:109727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665daa08) for lock 0x58a604dcef352321 00000080:00010000:2.0:1788572501.123918:0:109727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352321 from it ffff9e0382edbcd8 00010000:00010000:2.0:1788572501.123922:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572501.123931:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366600/0x58a604dcef352321 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792181 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572501.123960:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572501.123969:0:109727:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572501.123991:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572501.123995:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550d30, osc lock ffff92fc58371db0, flags 0x400001002 00010000:00010000:2.0:1788572501.124009:0:109727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245c00 type 11 mode 6 res 10737419265/551 (0 18446744073709551615) 00000100:00100000:1.0:1788572501.130606:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572501.130639:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:543b1dda-86fa-421e-9749-35cac38dec65:2393:1875449863835008:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572501.133747:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000080:00200000:1.0:1788572501.140605:0:109727:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:1.0:1788572501.140613:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70476750 (ffff92fc6721a1b8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572501.140617:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70476750 (ffff92fc6721a1b8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572501.140623:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550d30 (ffff92fc6721aea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572501.140627:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70476750 (ffff92fc6721a1b8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572501.140629:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550d30 (ffff92fc6721a1b8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572501.140642:0:109727:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572501.228967:0:109727:0:(file.c:5214:ll_file_seek()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], offset: 0, type: (null) 00000080:00200000:2.0:1788572501.228985:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572501.228995:0:109727:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572501.229046:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572501.229050:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550db0, osc lock ffff92fc58371c80, flags 0x400001002 00010000:00010000:2.0:1788572501.229071:0:109727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245c00 type 11 mode 6 res 10737419265/551 (0 18446744073709551615) 00010000:00010000:2.0:1788572501.229132:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.229147:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572501.229247:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788572501.229332:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572501.229354:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863835136:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572501.233580:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00010000:00010000:2.0:1788572501.235712:0:2395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235890:0:2395:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235908:0:2395:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235914:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235919:0:2395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572501.235929:0:2395:0:(osc_lock.c:187:osc_lock_lvb_update()) ### no dlm lock: acquired size 9000, setting rss=9000; leaving kms=9000, end=18446744073709551615 00000020:00200000:2.0:1788572501.235935:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572501.235948:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235954:0:2395:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235960:0:2395:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.235964:0:2395:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366e00 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:2.0:1788572501.235969:0:2395:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 00000000aeb69306/0x58a604dcef352367 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572501.236455:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572501.236470:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:543b1dda-86fa-421e-9749-35cac38dec65:2394:1875449863835264:192.168.202.132@tcp:23:lfs.0 00000020:00010000:2.0:1788572501.238578:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572501.238583:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572501.238588:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f26b0 (ffff92fc6721acf8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572501.238597:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 6/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.238608:0:109727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788572501.238613:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572501.238615:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f26b0 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572501.238629:0:109727:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572501.238638:0:109727:0:(file.c:5245:ll_file_seek()) COMPLETED file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_HOLE, rc = 12288 00000080:00200008:2.0:1788572501.238694:0:109727:0:(file.c:2483:do_file_read_iter()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], ppos: 0, count: 67108864 00000080:00200000:2.0:1788572501.238702:0:109727:0:(file.c:2090:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: read ppos: 0, bytes: 67108864 00000080:00200000:2.0:1788572501.238709:0:109727:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 0 00000020:00200000:2.0:1788572501.238713:0:109727:0:(cl_io.c:214:cl_io_rw_init()) header@00000000a81c5755[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572501.238715:0:109727:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:2.0:1788572501.238718:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572501.238767:0:109727:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:2.0:1788572501.238771:0:109727:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572501.238789:0:109727:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:2.0:1788572501.238801:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 81 00000008:00010000:2.0:1788572501.238806:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550530, osc lock ffff92fc583717c0, flags 0x140000 00010000:00010000:2.0:1788572501.238823:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 4/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.238842:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 67108863) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 4/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.238851:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 8/0,3 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.238858:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 8/0,3 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.238866:0:109727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 7/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572501.238874:0:109727:0:(vvp_io.c:835:vvp_io_read_start()) crhIBIW0-TeUOeG6ka7IzA: read [0, 67108864) 00000001:00010000:2.0:1788572501.238884:0:109727:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572501.238890:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572501.238892:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550c30, osc lock ffff92fc58370ab0, flags 0x400001002 00010000:00010000:2.0:1788572501.238899:0:109727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245c00 type 11 mode 6 res 10737419265/551 (0 18446744073709551615) 00010000:00010000:2.0:1788572501.238931:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35236e lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.238937:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000008b2dc00b/0x58a604dcef35236e lrc: 3/1,0 mode: --/PR res: [0x280000401:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572501.238961:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788572501.241227:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572501.241502:0:109666:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 2 min 2 max 34 00000100:00100000:1.0:1788572501.241509:0:109666:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_002' 00000080:00200000:2.0:1788572501.246529:0:109727:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572501.246538:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70474b90 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572501.246540:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70474b90 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572501.246545:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550c30 (ffff92fc6721acf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572501.246548:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70474b90 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572501.246550:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550c30 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572501.246559:0:109727:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:2.0:1788572501.246565:0:109727:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665daa08), size=67108864 (max 536870912), offset=0=0x0, pages 16384 (max 131072), locked, parallel 00000020:00200000:2.0:1788572501.246617:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000008:00010000:2.0:1788572501.246657:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000d7219bd7 ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 6/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.246687:0:107990:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 7/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00100000:2.0:1788572501.246746:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617f480 - object 0x280000401:551 offset 0<>12288 00000080:00200000:2.0:1788572501.250193:0:109727:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 12288 count 12288 file_offset 12288 00000020:00010000:2.0:1788572501.250208:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572501.250212:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572501.250217:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550530 (ffff92fc6721acf8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572501.250226:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(GROUP) ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 6/0,2 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.250237:0:109727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 5/0,1 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788572501.250246:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572501.250249:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550530 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572501.250257:0:109727:0:(vvp_io.c:533:vvp_io_advance()) advancing 12288 bytes 00000020:00200000:2.0:1788572501.250265:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572501.250272:0:109727:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572501.250277:0:109727:0:(file.c:2238:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: 1 io complete with rc: 0, result: 12288, restart: 0 00000020:00200000:2.0:1788572501.250292:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572501.250300:0:109727:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 12288 00000080:00200008:2.0:1788572501.250327:0:109727:0:(file.c:2650:do_file_write_iter()) START file . :VOLATILE:0000:339DA3E5:fd=03:[0x20000041e:0x1:0x0], ppos: 0, count: 12288 00000080:00200000:2.0:1788572501.250334:0:109727:0:(file.c:2090:ll_file_io_generic()) . :VOLATILE:0000:339DA3E5:fd=03: write ppos: 0, bytes: 12288 00000080:00200000:2.0:1788572501.250342:0:109727:0:(file.c:1848:ll_io_set_mirror()) . :VOLATILE:0000:339DA3E5:fd=03: desiginated mirror: 0 00000020:00200000:2.0:1788572501.250379:0:109727:0:(cl_io.c:214:cl_io_rw_init()) header@000000003491b48a[0x110, 2, [0x20000041e:0x1:0x0] hash] 00000020:00200000:2.0:1788572501.250382:0:109727:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 12288) 2048 0 00000080:00200000:2.0:1788572501.250387:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041e:0x1:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572501.250400:0:109727:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:2.0:1788572501.250407:0:109727:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 12288) 12288, 12288 00020000:00200000:2.0:1788572501.250410:0:109727:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572501.250426:0:109727:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 2] 00000008:00010000:2.0:1788572501.250438:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:2.0:1788572501.250442:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550530, osc lock ffff92fc583717c0, flags 0x40000 00000080:00200000:2.0:1788572501.250450:0:109727:0:(vvp_io.c:1292:vvp_io_write_start()) . :VOLATILE:0000:339DA3E5:fd=03: write [0, 12288) append 0 append lockpos 0 00000002:00010000:2.0:1788572501.250482:0:109727:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000041e:0x1:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572501.250536:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 3/1,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.250545:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 3/1,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.250556:0:109727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 3/1,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109727 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572501.250577:0:109727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863835648:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572501.250657:0:109727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.254445:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863835648:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572501.254474:0:109727:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 4/1,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.254490:0:109727:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 4/1,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.254501:0:109727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 4/1,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572501.254515:0:109727:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc4617d880 x1875449863835648/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 568/1072 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572501.254526:0:109727:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 3/1,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.254540:0:109727:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc671e4c00 type 13 mode 4 res 8589935646/1 (0 0) 00000080:00200000:2.0:1788572501.254547:0:109727:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000041e:0x1:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572501.254572:0:109727:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041e:0x1:0x0](ffff92fc665da1c8) for lock 0x58a604dcef352375 00000080:00010000:2.0:1788572501.254581:0:109727:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352375 from it ffff9e0382edb8e8 00010000:00010000:2.0:1788572501.254586:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 3/1,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.254596:0:109727:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 2/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572501.254642:0:109727:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041e:0x1:0x0](ffff92fc665da1c8), size=12288 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel 00000020:00200000:2.0:1788572501.254711:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:2.0:1788572501.254715:0:109727:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 12288 count 12288 file_offset 12288 00000080:00200000:2.0:1788572501.254766:0:109727:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041e:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788572501.254772:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572501.254775:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572501.254779:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550530 (ffff92fc6721acf8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572501.254782:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3a88 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572501.254785:0:109727:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550530 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572501.254794:0:109727:0:(vvp_io.c:533:vvp_io_advance()) advancing 12288 bytes 00000020:00200000:2.0:1788572501.254801:0:109727:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572501.254820:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc508a5840 00000008:00100000:2.0:1788572501.254972:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617d880 - object 0x2c0000401:537 offset 0<>12288 00000100:00100000:1.0:1788572501.262408:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:2.0:1788572501.263486:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:543b1dda-86fa-421e-9749-35cac38dec65:2396:1875449863835904:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572501.267414:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:543b1dda-86fa-421e-9749-35cac38dec65:2396:1875449863835904:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572501.280034:0:2395:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4990d800: committing for last_committed 4294968468 gen 1 00000008:00010000:2.0:1788572501.280097:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc508a5840 00000020:00200000:2.0:1788572501.280116:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:2.0:1788572501.280121:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:2.0:1788572501.280122:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1788572501.280131:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572501.280153:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc508a5840 00000100:00100000:2.0:1788572501.280195:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863835776:192.168.202.132@tcp:4:lfs.0 00000100:00100000:1.0:1788572501.293888:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:543b1dda-86fa-421e-9749-35cac38dec65:2394:1875449863836160:192.168.202.132@tcp:16:lfs.0 00000080:00200000:2.0:1788572501.296548:0:109727:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041e:0x1:0x0] updating i_size 12288 i_blocks 24 00000080:00200000:2.0:1788572501.296564:0:109727:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041e:0x1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572501.308248:0:109727:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:339DA3E5:fd=03:[0x20000041e:0x1:0x0](ffff92fc665da1c8) current size 12288, valid attrs 101f0, mode 92fc, uid 0, gid 0, new size 0, atime 1788572496.000000000, mtime 1788572496.000000000, ctime 1788572501.000000000 00010000:00010000:2.0:1788572501.308296:0:109727:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc51bc3e00/0x58a604dcef352344 lrc: 2/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x1a/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a2217921ce expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.308323:0:109727:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef352336 lrc: 2/0,0 mode: CR/CR res: [0x20000041e:0x1:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217921b9 expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572501.308387:0:109727:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc51bc3e00/0x58a604dcef352344 lrc: 1/0,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217921ce expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.308395:0:109727:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572501.308399:0:109727:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc51bc3e00/0x58a604dcef352344 lrc: 0/0,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217921ce expref: -99 pid: 109727 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572501.308494:0:109727:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572501.308508:0:109727:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d880 x1875449863836416/t0(0) o36->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572501.308536:0:109727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863836416:192.168.202.132@tcp:36:lfs.0 00000100:00100000:2.0:1788572501.308813:0:109727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.316164:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863836416:192.168.202.132@tcp:36:lfs.0 00000080:00200000:2.0:1788572501.316285:0:109727:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041e:0x1:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572501.316300:0:109727:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572501.316312:0:109727:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788572501.316324:0:109727:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:2.0:1788572501.316326:0:109727:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550530, osc lock ffff92fc583717c0, flags 0x0 00010000:00010000:2.0:1788572501.316394:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.316477:0:109727:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.316491:0:109727:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572501.316513:0:109727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863836544:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572501.316641:0:109727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.321424:0:109727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109727:1875449863836544:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572501.322820:0:109727:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.323355:0:109727:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.323387:0:109727:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572501.323407:0:109727:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 12288, setting rss=12288; kms=12288, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.323420:0:109727:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.323429:0:109727:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572501.327465:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863836672:192.168.202.132@tcp:10:lfs.0 00000100:00100000:1.0:1788572501.330646:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00010000:00010000:2.0:1788572501.331373:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572501.331381:0:109676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 2/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331390:0:109676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572501.331396:0:109676:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 2/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331403:0:109676:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 2/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331411:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 3/0,0 mode: PR/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331467:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 2/0,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331474:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572501.331508:0:109676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 1/0,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331515:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352375 lrc: 0/0,0 mode: --/PR res: [0x20000041e:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792206 expref: -99 pid: 109727 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.331531:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.331533:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.331534:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572501.331542:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572501.331554:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863837056:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572501.340587:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572501.340627:0:109665:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881860736 00000100:00100000:1.0:1788572501.340637:0:109665:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881860736 00000100:00100000:1.0:1788572501.340686:0:109665:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 9 00000100:00100000:1.0:1788572501.340696:0:109665:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:130896:x1875449881860736:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572501.340715:0:109665:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff92fc4990d800 lock: 0000000007569468/0x58a604dcef35237c lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a22179221b expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572501.340826:0:109665:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572501.340834:0:109665:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:130896:x1875449881860736:12345-192.168.202.132@tcp:104: Request processed in 139us (258us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572501.340842:0:109665:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 9 00000100:00100000:2.0:1788572501.349083:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:543b1dda-86fa-421e-9749-35cac38dec65:2395:1875449863837056:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572501.371678:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572501.371686:0:109676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 2/0,0 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x800020400020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.371694:0:109676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2200 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:2.0:1788572501.371700:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 3/0,0 mode: GROUP/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x800028400020000 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572501.371757:0:109676:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572501.371769:0:109676:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572501.371802:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 2/0,0 mode: --/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x804869400020020 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.371810:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572501.371837:0:109676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 1/0,0 mode: --/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x804869400020020 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.371846:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc4990d800 lock: 000000000268bc32/0x58a604dcef352352 lrc: 0/0,0 mode: --/GROUP res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 881528466 flags: 0x804869400020020 nid: local remote: 0xc9a239a2217921dc expref: -99 pid: 109727 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.371877:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.371879:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.371880:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572501.371889:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572501.371901:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:543b1dda-86fa-421e-9749-35cac38dec65:2396:1875449863837952:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572501.374234:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:543b1dda-86fa-421e-9749-35cac38dec65:2396:1875449863837952:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572501.421956:0:109729:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109729:1875449863839232:192.168.202.132@tcp:35:lfs.0 00000002:00100000:1.0:1788572501.422086:0:109729:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67048a80 x1875449863838848/t4294983678(4294983678) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572501.444705:0:109730:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8),name=/ 00000002:00010000:2.0:1788572501.444835:0:109730:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572501.444978:0:109730:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572501.444999:0:109730:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572501.445008:0:109730:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572501.445017:0:109730:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:2.0:1788572501.445021:0:109730:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef35230c 00000080:00010000:2.0:1788572501.445029:0:109730:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35230c from it ffff9e038237bbf8 00010000:00010000:2.0:1788572501.445033:0:109730:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572501.445043:0:109730:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572501.445060:0:109730:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572501.445070:0:109730:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:2.0:1788572501.445076:0:109730:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d21c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572501.445089:0:109730:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d21c8), file ffff92fc4711b680, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572501.445096:0:109730:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d21c8), intent=open 00000002:00010000:2.0:1788572501.445284:0:109730:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572501.445340:0:109730:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.445349:0:109730:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572501.445979:0:109730:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572501.445995:0:109730:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e300 x1875449863839488/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572501.446007:0:109730:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572501.446031:0:109730:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109730:1875449863839488:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572501.446575:0:109730:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.453755:0:109730:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109730:1875449863839488:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572501.453783:0:109730:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453797:0:109730:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453804:0:109730:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453811:0:109730:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453820:0:109730:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453825:0:109730:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572501.453830:0:109730:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453836:0:109730:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453843:0:109730:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.453850:0:109730:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572501.454069:0:109730:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.454081:0:109730:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366200/0x58a604dcef352398 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572501.454116:0:109730:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc4617e300 x1875449863839488/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572501.454131:0:109730:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572501.454258:0:109730:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8),name=/ 00000002:00010000:2.0:1788572501.454280:0:109730:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572501.454295:0:109730:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572501.454307:0:109730:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572501.454315:0:109730:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572501.454321:0:109730:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:2.0:1788572501.454324:0:109730:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef35230c 00000080:00010000:2.0:1788572501.454330:0:109730:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35230c from it ffff9e038237bbf8 00010000:00010000:2.0:1788572501.454333:0:109730:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572501.454340:0:109730:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572501.454354:0:109730:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572501.454372:0:109730:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:2.0:1788572501.454380:0:109730:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d21c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572501.454388:0:109730:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d21c8), flags 304000 00000002:00010000:2.0:1788572501.454404:0:109730:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240304001 00010000:00010000:2.0:1788572501.454574:0:109730:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef35239f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572501.454583:0:109730:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef35239f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572501.454659:0:109730:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572501.454667:0:109730:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e300 x1875449863839744/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572501.454677:0:109730:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef35239f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109730 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572501.454697:0:109730:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:543b1dda-86fa-421e-9749-35cac38dec65:109730:1875449863839744:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572501.697768:0:109735:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:2.0:1788572501.697816:0:109735:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc4990d800 00010000:00010000:2.0:1788572501.697838:0:109735:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572501.697852:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572501.697929:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572501.697966:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef35239f lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217922b5 expref: -99 pid: 109730 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:2.0:1788572501.698192:0:109676:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617e300 x1875449863839744/t4294983680(4294983680) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572517 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572501.698204:0:109676:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4990d800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572501.698211:0:109676:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c000 x1875449863840128/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572501.698236:0:109676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:543b1dda-86fa-421e-9749-35cac38dec65:109676:1875449863840128:192.168.202.132@tcp:35:ldlm_bl.0 00010000:00010000:1.0:1788572501.698394:0:109732:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc4990d800 00010000:00010000:1.0:1788572501.698421:0:109732:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc4990d800 00010000:00010000:1.0:1788572501.698440:0:109732:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc4990d800 00000100:00100000:2.0:1788572501.701510:0:109676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.735512:0:109676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:543b1dda-86fa-421e-9749-35cac38dec65:109676:1875449863840128:192.168.202.132@tcp:35:ldlm_bl.0 00000002:00100000:2.0:1788572501.735551:0:109676:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617e300 x1875449863839744/t4294983680(4294983680) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572501.735626:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc51bc3800/0x58a604dcef352383 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179225a expref: -99 pid: 109729 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.735671:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572501.735680:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef35239f lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217922b5 expref: -99 pid: 109730 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572501.735687:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc51bc3800/0x58a604dcef352383 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179225a expref: -99 pid: 109729 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.735692:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:2.0:1788572501.735710:0:109676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:543b1dda-86fa-421e-9749-35cac38dec65:109676:1875449863840256:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572501.735850:0:109676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e3c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572501.757280:0:109676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:543b1dda-86fa-421e-9749-35cac38dec65:109676:1875449863840256:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572501.757309:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc702f5600/0x58a604dcef35230c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179213b expref: -99 pid: 109726 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572501.757339:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70366800/0x58a604dcef35239f lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217922b5 expref: -99 pid: 109730 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572501.757348:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc51bc3800/0x58a604dcef352383 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179225a expref: -99 pid: 109729 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572501.757407:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.757410:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572501.757412:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572501.965398:0:109387:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041e:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572501.965418:0:109387:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572501.965451:0:109387:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041e:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572501.965460:0:109387:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041e:0x1:0x0](ffff92fc665da1c8) 00000080:00200000:2.0:1788572501.965602:0:109387:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d21c8) 00000080:00200000:2.0:1788572501.965620:0:109387:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572501.965626:0:109387:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572501.965636:0:109387:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572501.965640:0:109387:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665daa08) 00000100:00100000:2.0:1788572502.905110:0:109764:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job lsof:543b1dda-86fa-421e-9749-35cac38dec65:109764:1875449863840384:192.168.202.132@tcp:101:lsof.0 00010000:00010000:2.0:1788572502.905139:0:109764:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905151:0:109764:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905163:0:109764:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905175:0:109764:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905187:0:109764:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905195:0:109764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905202:0:109764:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572502.905214:0:109764:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733b800 x1875449863840384/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572518 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572502.905228:0:109764:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905241:0:109764:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc671e4c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572502.905248:0:109764:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572502.905278:0:109764:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572502.905285:0:109764:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef3523a6 00000080:00010000:2.0:1788572502.905290:0:109764:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:2.0:1788572502.905293:0:109764:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3523a6 from it ffff9e0382f4bcb8 00010000:00010000:2.0:1788572502.905296:0:109764:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572502.905311:0:109764:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572503.684858:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572504.624804:0:109772:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665de3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572504.624818:0:109772:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8),name=/ 00000002:00010000:2.0:1788572504.626149:0:109772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572504.626176:0:109772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572504.626194:0:109772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572504.626203:0:109772:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572504.626214:0:109772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:2.0:1788572504.626219:0:109772:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef3523a6 00000080:00010000:2.0:1788572504.626228:0:109772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3523a6 from it ffff9e0382f93cb8 00010000:00010000:2.0:1788572504.626233:0:109772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572504.626242:0:109772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788572504.626466:0:109772:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665de3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572504.626472:0:109772:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8),name=/ 00000002:00010000:2.0:1788572504.626485:0:109772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572504.626493:0:109772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572504.626502:0:109772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572504.626510:0:109772:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572504.626513:0:109772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00010000:2.0:1788572504.626516:0:109772:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665de3c8) for lock 0x58a604dcef3523a6 00000080:00010000:2.0:1788572504.626519:0:109772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3523a6 from it ffff9e0382f93cb8 00010000:00010000:2.0:1788572504.626523:0:109772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572504.626530:0:109772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572504.648456:0:109772:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665de3c8) 00000080:00200000:2.0:1788572504.648498:0:109772:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc4990d800 (sb ffff92fc4990d800) 00010000:00010000:2.0:1788572504.648540:0:109772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648552:0:109772:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648560:0:109772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648567:0:109772:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648593:0:109772:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572504.648599:0:109772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648606:0:109772:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648613:0:109772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648619:0:109772:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648636:0:109772:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1788572504.648641:0:109772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648648:0:109772:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648654:0:109772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648660:0:109772:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572504.648666:0:109772:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:2.0:1788572504.648669:0:109772:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:1.0:1788572504.648925:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788572504.648933:0:109676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.648941:0:109676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f5400 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572504.648945:0:109676:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.648955:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572504.648972:0:109676:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572504.648978:0:109676:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572504.648980:0:109676:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:1.0:1788572504.648983:0:109676:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:1.0:1788572504.649020:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649026:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572504.649053:0:109676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649066:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef3522a3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792103 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649077:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572504.649080:0:109676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649085:0:109676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2c00 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572504.649088:0:109676:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649094:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572504.649101:0:109676:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572504.649105:0:109676:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572504.649107:0:109676:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:1.0:1788572504.649109:0:109676:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572504.649117:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649122:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572504.649127:0:109676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649133:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3522d4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179210a expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649147:0:109676:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572504.649150:0:109676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649156:0:109676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2000 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572504.649159:0:109676:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649165:0:109676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572504.649170:0:109676:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572504.649175:0:109676:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572504.649177:0:109676:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:1.0:1788572504.649178:0:109676:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:1.0:1788572504.649185:0:109676:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649190:0:109676:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572504.649194:0:109676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649200:0:109676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3522db lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792111 expref: -99 pid: 109664 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572504.649207:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572504.649209:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572504.649211:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572504.649228:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9e588565-c27f-4f6e-85de-718639cb4969:2394:1875449863840512:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572504.649320:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9e588565-c27f-4f6e-85de-718639cb4969:2394:1875449863840768:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572504.649341:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572504.649359:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:9e588565-c27f-4f6e-85de-718639cb4969:2394:1875449863840640:192.168.202.132@tcp:103:ldlm_bl.0 00020000:01000000:2.0:1788572504.650293:0:109772:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc5838d800 active: 1 00020000:01000000:2.0:1788572504.650299:0:109772:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc5838a800 active: 1 00020000:01000000:2.0:1788572504.650302:0:109772:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788572504.650307:0:109772:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc4990d800 00020000:01000000:2.0:1788572504.650309:0:109772:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc4990d800: disconnecting target lustre-OST0001-osc-ffff92fc4990d800 00000100:00080000:2.0:1788572504.650319:0:109772:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 543b1dda-86fa-421e-9749-35cac38dec65->lustre-OST0001_UUID 00000100:00080000:2.0:1788572504.652321:0:109772:0:(import.c:65:import_set_state_nolock()) ffff92fc5838d000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572504.653126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863840896:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572504.668547:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5838d000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572504.668577:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863840896:192.168.202.132@tcp:9:umount.0 00000100:00080000:2.0:1788572504.670827:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572504.670835:0:109772:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4990d800: committing for last_committed 4294968468 gen 2 00000100:00100000:2.0:1788572504.670909:0:109772:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617d500 x1875449863836672/t4294968469(4294968469) o10->lustre-OST0001-osc-ffff92fc4990d800@192.168.202.132@tcp:6/4 lens 440/432 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572504.670937:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572504.672205:0:109772:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352305 00000020:00000080:2.0:1788572504.672230:0:109772:0:(genops.c:885:class_export_put()) final put ffff92fc5838a800/LOV_OSC_UUID 00000020:01000000:2.0:1788572504.672293:0:109772:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc4990d800 (flags='') 00000020:00000080:2.0:1788572504.672299:0:109772:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572504.672308:0:109772:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc4990d800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572504.672312:0:109772:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc51cdcd40) has no exports 00000020:00000080:1.0:1788572504.672711:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838a800/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc4990d800 00000008:01000000:2.0:1788572504.678086:0:109772:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc4990d800: client import never connected 00000100:00080000:2.0:1788572504.678095:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572504.678099:0:109772:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4990d800: committing for last_committed 4294968468 gen 3 00000100:00100000:2.0:1788572504.678107:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572504.678316:0:109772:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572504.678328:0:109772:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc4990d800 (uuid 543b1dda-86fa-421e-9749-35cac38dec65) 00020000:01000000:2.0:1788572504.678334:0:109772:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc4990d800 00020000:01000000:2.0:1788572504.678337:0:109772:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc4990d800: disconnecting target lustre-OST0000-osc-ffff92fc4990d800 00000100:00080000:2.0:1788572504.678366:0:109772:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 543b1dda-86fa-421e-9749-35cac38dec65->lustre-OST0000_UUID 00000100:00080000:2.0:1788572504.678477:0:109772:0:(import.c:65:import_set_state_nolock()) ffff92fc5838a000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:1.0:1788572504.678987:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5838d000 for lustre-OST0001-osc-ffff92fc4990d800 00000020:00000080:1.0:1788572504.678996:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc58389800/543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:1.0:1788572504.678999:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58389800/543b1dda-86fa-421e-9749-35cac38dec65 for lustre-OST0001-osc-ffff92fc4990d800 00000020:01000000:1.0:1788572504.679005:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc4990d800 (543b1dda-86fa-421e-9749-35cac38dec65) 00000100:00080000:2.0:1788572504.691439:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572504.691445:0:109772:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4990d800: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572504.691451:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572504.691584:0:109772:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3522fe 00000020:00000080:2.0:1788572504.691603:0:109772:0:(genops.c:885:class_export_put()) final put ffff92fc5838d800/LOV_OSC_UUID 00000020:01000000:2.0:1788572504.691668:0:109772:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc4990d800 (flags='') 00000020:00000080:2.0:1788572504.691674:0:109772:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572504.691683:0:109772:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc4990d800: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572504.691687:0:109772:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc51cdb380) has no exports 00000020:00000080:1.0:1788572504.691943:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838d800/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc4990d800 00000008:01000000:2.0:1788572504.692103:0:109772:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc4990d800: client import never connected 00000100:00080000:2.0:1788572504.692107:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572504.692109:0:109772:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4990d800: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572504.692113:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572504.692295:0:109772:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572504.692302:0:109772:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc4990d800 (uuid 543b1dda-86fa-421e-9749-35cac38dec65) 00000020:00000080:2.0:1788572504.692306:0:109772:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3522f7 00000020:00000080:2.0:1788572504.692312:0:109772:0:(genops.c:885:class_export_put()) final put ffff92fc58389000/543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:1.0:1788572504.710315:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5838a000 for lustre-OST0000-osc-ffff92fc4990d800 00000020:00000080:1.0:1788572504.710334:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc5838c800/543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:1.0:1788572504.710337:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838c800/543b1dda-86fa-421e-9749-35cac38dec65 for lustre-OST0000-osc-ffff92fc4990d800 00000020:01000000:1.0:1788572504.710373:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc4990d800 (543b1dda-86fa-421e-9749-35cac38dec65) 00010000:00010000:1.0:1788572504.711235:0:109676:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572504.716267:0:109772:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 543b1dda-86fa-421e-9749-35cac38dec65->lustre-MDT0000_UUID 00010000:00010000:1.0:1788572504.716405:0:109772:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572504.716476:0:109772:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572504.716483:0:109772:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572504.716503:0:109772:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678480 going to sleep for 0 seconds 00000100:00100000:1.0:1788572504.716515:0:109772:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job umount:543b1dda-86fa-421e-9749-35cac38dec65:109772:1875449863841152:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572504.716526:0:109772:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572504.716535:0:109772:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572504.716540:0:109772:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4990d800 lock: ffff92fc70204c00/0x58a604dcef3523a6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217922df expref: -99 pid: 109764 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00080000:1.0:1788572504.717795:0:109772:0:(import.c:65:import_set_state_nolock()) ffff92fc4a3b7000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572504.718981:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863841280:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572504.724303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4a3b7000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572504.724339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863841280:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572504.724417:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572504.724422:0:109772:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4990d800: committing for last_committed 4294983670 gen 2 00000100:00100000:1.0:1788572504.724439:0:109772:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617d880 x1875449863836416/t4294983671(4294983671) o36->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572504.724463:0:109772:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67338000 x1875449863837568/t4294983674(4294983674) o61->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 584/224 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572504.724474:0:109772:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6733b480 x1875449863838208/t4294983677(4294983677) o35->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572517 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572504.724485:0:109772:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6728b100 x1875449863841280/t4294983682(4294983682) o39->lustre-MDT0000-mdc-ffff92fc4990d800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572514 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572504.724498:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572504.726310:0:109772:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3522e9 00000020:00000080:1.0:1788572504.726326:0:109772:0:(genops.c:885:class_export_put()) final put ffff92fc5df7b800/543b1dda-86fa-421e-9749-35cac38dec65 00000100:00080000:1.0:1788572504.726436:0:109772:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 543b1dda-86fa-421e-9749-35cac38dec65->lustre-MDT0001_UUID 00000100:00100000:2.0:1788572504.730568:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863841408:192.168.202.132@tcp:39:umount.0 00000100:00100000:2.0:1788572504.748900:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4990d800: committing for last_committed 4294976745 gen 1 00000100:00080000:2.0:1788572504.748919:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58388000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572504.748950:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:543b1dda-86fa-421e-9749-35cac38dec65:2392:1875449863841408:192.168.202.132@tcp:39:umount.0 00000020:00000080:2.0:1788572504.758921:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df7d800/543b1dda-86fa-421e-9749-35cac38dec65 for lustre-MDT0001-mdc-ffff92fc4990d800 00000020:00000080:2.0:1788572504.758949:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df79800/543b1dda-86fa-421e-9749-35cac38dec65 for lustre-clilmv-ffff92fc4990d800 00000020:00000080:2.0:1788572504.758957:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc455d9800/543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:2.0:1788572504.758960:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc455d9800/543b1dda-86fa-421e-9749-35cac38dec65 for lustre-clilmv-ffff92fc4990d800 00000020:01000000:2.0:1788572504.758964:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc4990d800 (543b1dda-86fa-421e-9749-35cac38dec65) 00000020:00000080:1.0:1788572504.769269:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4a3b7000 for lustre-MDT0000-mdc-ffff92fc4990d800 00000020:00000080:1.0:1788572504.769282:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc455dd000/543b1dda-86fa-421e-9749-35cac38dec65 00000020:00000080:1.0:1788572504.769285:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc455dd000/543b1dda-86fa-421e-9749-35cac38dec65 for lustre-MDT0000-mdc-ffff92fc4990d800 00000020:01000000:1.0:1788572504.769297:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc4990d800 (543b1dda-86fa-421e-9749-35cac38dec65) 00000100:00100000:2.0:1788572504.786593:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9e588565-c27f-4f6e-85de-718639cb4969:2392:1875449863841664:192.168.202.132@tcp:251:umount.0 00000100:00080000:2.0:1788572504.796285:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572504.796294:0:109772:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572504.796299:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572504.796302:0:109772:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572504.796314:0:109772:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352295 00000020:00000080:2.0:1788572504.796324:0:109772:0:(genops.c:885:class_export_put()) final put ffff92fc492a6800/9e588565-c27f-4f6e-85de-718639cb4969 00000020:01000000:2.0:1788572504.796354:0:109772:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:2.0:1788572504.796366:0:109772:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572504.796374:0:109772:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572504.796379:0:109772:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc48cc8000) has no exports 10000000:01000000:1.0:1788572504.798645:0:109678:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788572504.798671:0:109772:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:2.0:1788572504.798677:0:109772:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572504.798684:0:109772:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572504.798692:0:109772:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572504.798695:0:109772:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572504.798847:0:109772:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572504.798855:0:109772:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 9e588565-c27f-4f6e-85de-718639cb4969) 00000020:01000004:2.0:1788572504.798861:0:109772:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc4990d800 1 00000020:01000004:2.0:1788572504.798864:0:109772:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc4990f800 00000100:00100000:1.0:1788572504.806787:0:109665:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc51201200:109665] 0 exiting: rc = 1 00000100:00100000:1.0:1788572504.806848:0:109666:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc46538a00:109666] 1 exiting: rc = 0 00000100:00100000:1.0:1788572504.822999:0:109669:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc46538c00:109669] 0 exiting: rc = 1 00000100:00100000:2.0:1788572505.450108:0:109785:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 10000000:01000000:2.0:1788572505.474396:0:109799:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000100:00100000:2.0:1788572505.479019:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:2392:1875449863841792:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572505.490811:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572505.490832:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:2.0:1788572505.490843:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:2.0:1788572505.490849:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:2.0:1788572505.490854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:2.0:1788572505.490858:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572505.490870:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:2.0:1788572505.490884:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000088dd104f): req@ffff92fc67048e00 x1875449863841920/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572505.490905:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:2392:1875449863841792:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572505.491028:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce50c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.498123:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863841920:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572505.498158:0:109785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.498172:0:109785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572505.498312:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863842048:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572505.498404:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce50c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.517535:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863842048:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:2.0:1788572505.517588:0:109785:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572505.517613:0:109785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.517625:0:109785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572505.517634:0:109785:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc53e08000 10000000:01000000:2.0:1788572505.517641:0:109785:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:2.0:1788572505.517645:0:109785:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc53e08000 10000000:01000000:2.0:1788572505.517649:0:109785:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572505.517652:0:109785:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc53e08000 10000000:01000000:2.0:1788572505.517654:0:109785:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:2.0:1788572505.517657:0:109785:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc53e08000 from 1 10000000:01000000:2.0:1788572505.517660:0:109785:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788572505.517697:0:109785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366c00/0x58a604dcef3523c2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.517706:0:109785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366c00/0x58a604dcef3523c2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.517715:0:109785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366c00/0x58a604dcef3523c2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572505.517763:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863842176:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572505.517850:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce50c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.521888:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863842176:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572505.521924:0:109785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366c00/0x58a604dcef3523c2 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217922f4 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.521939:0:109785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366c00/0x58a604dcef3523c2 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217922f4 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572505.522023:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863842304:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572505.522107:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce50c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572505.604099:0:109803:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109803:1875449863842688:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572505.604146:0:109803:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572505.604159:0:109803:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572505.604161:0:109803:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788572505.604166:0:109803:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788572505.604208:0:109803:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109803:1875449863842816:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572505.604377:0:109803:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc70678600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572505.617287:0:109803:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109803:1875449863842816:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572505.617337:0:109803:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572505.622919:0:109803:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572505.622927:0:109803:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:1.0:1788572505.622933:0:109803:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000100:00100000:2.0:1788572505.630100:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48e0eb40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.632896:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843072:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572505.633608:0:109785:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572505.633635:0:109785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef3523f3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217922fb expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.633647:0:109785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef3523f3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217922fb expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572505.633749:0:109785:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc53e08000 from 1 10000000:01000000:2.0:1788572505.633754:0:109785:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572505.633796:0:109785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.633804:0:109785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.633815:0:109785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572505.633843:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843200:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572505.633928:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.642437:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843200:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572505.642467:0:109785:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792302 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.642480:0:109785:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792302 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572505.642563:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843328:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572505.642650:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.646833:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843328:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572505.646904:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843456:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572505.647005:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.659797:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109785:1875449863843456:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572505.661276:0:109809:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109809:1875449863843584:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572505.661393:0:109809:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.664592:0:109809:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109809:1875449863843584:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572505.664635:0:109809:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572505.664648:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572505.664650:0:109809:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572505.664671:0:109809:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc53e08000 00000020:00000080:2.0:1788572505.664674:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572505.664684:0:109809:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572505.664686:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572505.664688:0:109809:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572505.664690:0:109809:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572505.664692:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572505.664694:0:109809:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572505.664697:0:109809:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc53e08000 00000020:00000080:2.0:1788572505.664699:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572505.664793:0:109809:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572505.664797:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572505.664798:0:109809:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:2.0:1788572505.664801:0:109809:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572505.664804:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572505.664805:0:109809:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:2.0:1788572505.664808:0:109809:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc53e08000 00000020:00000080:2.0:1788572505.664810:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:2.0:1788572505.671416:0:109809:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 6596us 00000100:00100000:2.0:1788572505.671503:0:109809:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109809:1875449863843712:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572505.671628:0:109809:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.676705:0:109809:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109809:1875449863843712:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572505.676767:0:109809:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572505.676776:0:109809:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572505.676778:0:109809:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:2.0:1788572505.676782:0:109809:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:1.0:1788572505.682197:0:109785:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:1.0:1788572505.682232:0:109785:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788572505.682262:0:109785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792302 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572505.682275:0:109785:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef3523fa lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792302 expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788572505.682297:0:109785:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000100:00100000:1.0:1788572505.708251:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fdebee5e-f7b1-4f7d-9968-d622b2cde851:0:1875449863844224:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572505.708365:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109785:1875449863844480:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:2.0:1788572505.708384:0:109785:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:2.0:1788572505.708399:0:109785:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:2.0:1788572505.708404:0:109785:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:2.0:1788572505.708416:0:109785:0:(genops.c:1385:class_connect()) connect: client fdebee5e-f7b1-4f7d-9968-d622b2cde851, cookie 0x58a604dcef352416 00000020:00000080:2.0:1788572505.708438:0:109785:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef35241d 00000100:00080000:2.0:1788572505.708446:0:109785:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc53e08000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572505.708455:0:109785:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc53e08000: import ffff92fc46150800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572505.708469:0:109785:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572505.708494:0:109785:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449863844608/t0(0) o8->lustre-OST0000-osc-ffff92fc53e08000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572505.708521:0:109785:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import fdebee5e-f7b1-4f7d-9968-d622b2cde851->lustre-OST0000_UUID 00020000:01000000:2.0:1788572505.708540:0:109785:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc53e08000) active 00000020:00000080:2.0:1788572505.708574:0:109785:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352424 00000100:00080000:2.0:1788572505.708581:0:109785:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc53e08000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572505.708586:0:109785:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc53e08000: import ffff92fc46155000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572505.708592:0:109785:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572505.708605:0:109785:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863844736/t0(0) o8->lustre-OST0001-osc-ffff92fc53e08000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572505.708615:0:109785:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import fdebee5e-f7b1-4f7d-9968-d622b2cde851->lustre-OST0001_UUID 00020000:01000000:2.0:1788572505.708622:0:109785:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc53e08000) active 00000080:02000400:2.0:1788572505.708632:0:109785:0:(llite_lib.c:801:client_common_fill_super()) Test dummy encryption mode enabled 00000100:00100000:2.0:1788572505.708679:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109785:1875449863844864:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788572505.708817:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572505.723974:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:fdebee5e-f7b1-4f7d-9968-d622b2cde851:0:1875449863843968:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572505.724109:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109785:1875449863844864:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788572505.724175:0:109785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109785:1875449863844992:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:2.0:1788572505.724305:0:109785:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.737782:0:109785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109785:1875449863844992:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:2.0:1788572505.737964:0:109785:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e2a08) 00000080:00200000:2.0:1788572505.737973:0:109785:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:2.0:1788572505.738038:0:109785:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:2.0:1788572505.738042:0:109785:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00000080:00200000:1.0:1788572505.928118:0:109827:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e2a08),name=/ 00000002:00010000:1.0:1788572505.928149:0:109827:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572505.928214:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572505.928225:0:109827:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572505.928236:0:109827:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572505.928261:0:109827:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845120:192.168.202.132@tcp:101:cmp.0 00000100:00100000:1.0:1788572505.929061:0:109827:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce5480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.932102:0:109827:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845120:192.168.202.132@tcp:101:cmp.0 00010000:00010000:2.0:1788572505.932133:0:109827:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932144:0:109827:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932155:0:109827:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932171:0:109827:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932183:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932191:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932197:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572505.932210:0:109827:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b100 x1875449863845120/t0(0) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572521 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572505.932224:0:109827:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932239:0:109827:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc602d5800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572505.932247:0:109827:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572505.932279:0:109827:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572505.932284:0:109827:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) for lock 0x58a604dcef35242b 00000080:00010000:2.0:1788572505.932289:0:109827:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572505.932293:0:109827:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35242b from it ffff9e0382ed3bf8 00010000:00010000:2.0:1788572505.932296:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932310:0:109827:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572505.932335:0:109827:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e2a08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572505.932368:0:109827:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e2a08), flags=25 00000080:00200000:2.0:1788572505.932373:0:109827:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e2a08), intent=0 00000002:00010000:2.0:1788572505.932527:0:109827:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572505.932582:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932590:0:109827:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.932598:0:109827:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572505.932619:0:109827:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845248:192.168.202.132@tcp:101:cmp.0 00000100:00100000:2.0:1788572505.932694:0:109827:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572505.939273:0:109827:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845248:192.168.202.132@tcp:101:cmp.0 00010000:00010000:2.0:1788572505.939301:0:109827:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939324:0:109827:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939333:0:109827:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939353:0:109827:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939365:0:109827:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939378:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939452:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939459:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572505.939472:0:109827:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617e300 x1875449863845248/t0(0) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572521 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572505.939485:0:109827:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939495:0:109827:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc602d5800 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:2.0:1788572505.939501:0:109827:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572505.939531:0:109827:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e21c8) 00000080:00200000:2.0:1788572505.939539:0:109827:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572505.939686:0:109827:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e21c8) for lock 0x58a604dcef352432 00000080:00010000:2.0:1788572505.939699:0:109827:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e21c8) 00000080:00010000:2.0:1788572505.939704:0:109827:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352432 from it ffff9e0382ed3ac8 00010000:00010000:2.0:1788572505.939707:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572505.939717:0:109827:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572505.939814:0:109827:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e21c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572505.939833:0:109827:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f59b.sanity-sec, dir=[0x20000041d:0x1:0x0](ffff92fc665e21c8), file ffff92fc47c71380, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1788572505.939840:0:109827:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f59b.sanity-sec, dir=[0x20000041d:0x1:0x0](ffff92fc665e21c8), intent=open 00000100:00100000:2.0:1788572506.123075:0:109827:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845376:192.168.202.132@tcp:900:cmp.0 00000100:00100000:2.0:1788572506.123544:0:109827:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572506.128127:0:109827:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845376:192.168.202.132@tcp:900:cmp.0 00000002:00010000:2.0:1788572506.128169:0:109827:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1788572506.128234:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.128244:0:109827:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572506.128255:0:109827:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc53e08000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572506.128264:0:109827:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e300 x1875449863845632/t0(0) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572506.128273:0:109827:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109827 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572506.128293:0:109827:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845632:192.168.202.132@tcp:101:cmp.0 00000100:00100000:2.0:1788572506.128829:0:109827:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572506.150652:0:109827:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc53e08000: committing for last_committed 4294983683 gen 1 00000100:00100000:2.0:1788572506.150674:0:109827:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617e300 x1875449863845632/t4294983683(4294983683) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 776/696 e 0 to 0 dl 1788572522 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572506.150688:0:109827:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863845632:192.168.202.132@tcp:101:cmp.0 00010000:00010000:2.0:1788572506.150715:0:109827:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150780:0:109827:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150790:0:109827:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150802:0:109827:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150815:0:109827:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150831:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150839:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150847:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572506.150859:0:109827:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617e300 x1875449863845632/t4294983683(4294983683) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 776/696 e 0 to 0 dl 1788572522 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572506.150872:0:109827:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e300 x1875449863845632/t4294983683(4294983683) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 776/696 e 0 to 0 dl 1788572522 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572506.150885:0:109827:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572506.150900:0:109827:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.150910:0:109827:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc602d5800 type 13 mode 4 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572506.150918:0:109827:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572506.150960:0:109827:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e42c8) 00000080:00200000:2.0:1788572506.151067:0:109827:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:2.0:1788572506.151082:0:109827:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e42c8) for lock 0x58a604dcef352439 00000080:00010000:2.0:1788572506.151166:0:109827:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e42c8) 00000080:00200008:2.0:1788572506.151181:0:109827:0:(file.c:1090:ll_file_open()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665e42c8), flags 100000 00000002:00100000:2.0:1788572506.151202:0:109827:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e300 x1875449863845632/t4294983683(4294983683) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 592/696 e 0 to 0 dl 1788572522 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572506.151250:0:109827:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352439 from it ffff92fc51c92b40 00010000:00010000:2.0:1788572506.151255:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.151265:0:109827:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572506.151292:0:109827:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e42c8), inode mode 81a4 mask 44 00000080:00200008:2.0:1788572506.151313:0:109827:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665e42c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572506.151318:0:109827:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e42c8),name=f59b.sanity-sec 00000002:00010000:2.0:1788572506.151356:0:109827:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572506.151365:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.151378:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572506.151385:0:109827:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572506.151390:0:109827:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e42c8) 00000080:00010000:2.0:1788572506.151393:0:109827:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e42c8) for lock 0x58a604dcef352439 00000080:00010000:2.0:1788572506.151396:0:109827:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352439 from it ffff9e0382ed3cd8 00010000:00010000:2.0:1788572506.151399:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.151406:0:109827:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572506.151422:0:109827:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572506.151432:0:109827:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572506.151468:0:109827:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572506.151472:0:109827:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550330, osc lock ffff92fc58370ab0, flags 0x400001002 00010000:00010000:2.0:1788572506.151516:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572506.151530:0:109827:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572506.151573:0:109827:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1788572506.157781:0:109827:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572506.157792:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572506.157797:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.157803:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550330 (ffff92fc6721abd8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.157808:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572506.157811:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550330 (ffff92fc6721a3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572506.157826:0:109827:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572506.157943:0:109827:0:(file.c:5214:ll_file_seek()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:2.0:1788572506.157952:0:109827:0:(file.c:5245:ll_file_seek()) COMPLETED file f59b.sanity-sec:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_CUR, rc = 0 00000080:00200008:2.0:1788572506.157993:0:109827:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 0, count: 4194304 00000080:00200000:2.0:1788572506.158012:0:109827:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:2.0:1788572506.158047:0:109827:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 0, bytes: 4194304 00000080:00200000:2.0:1788572506.158053:0:109827:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 0 00000020:00200000:2.0:1788572506.158059:0:109827:0:(cl_io.c:214:cl_io_rw_init()) header@00000000204bb37a[0x110, 1, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572506.158060:0:109827:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:2.0:1788572506.158065:0:109827:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1788572506.158081:0:109827:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:2.0:1788572506.158085:0:109827:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572506.158102:0:109827:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:2.0:1788572506.158113:0:109827:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572506.158117:0:109827:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550330, osc lock ffff92fc58370ab0, flags 0x0 00010000:00010000:2.0:1788572506.158137:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158159:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158172:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158181:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158189:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572506.158198:0:109827:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [0, 4194304) 00000001:00010000:2.0:1788572506.158209:0:109827:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572506.158214:0:109827:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572506.158216:0:109827:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550b30, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572506.158222:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158232:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572506.158239:0:109827:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572506.158243:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158256:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572506.158262:0:109827:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572506.158266:0:109827:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572506.158269:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572506.158272:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.158274:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550b30 (ffff92fc6721abd8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.158277:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572506.158279:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550b30 (ffff92fc6721a3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572506.158285:0:109827:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00010000:00010000:2.0:1788572506.158322:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158333:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (4096 8191) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158392:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/3,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.158402:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (12288 16383) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/3,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572506.159507:0:109827:0:(osc_cache.c:3059:osc_queue_sync_pages()) ### no dlm lock: extent: ffff92fc451f56c0 00010000:00010000:2.0:1788572506.159536:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 8/3,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.159600:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.159611:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.159618:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572506.159674:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc451f56c0 00010000:00010000:2.0:1788572506.159699:0:107990:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:2.0:1788572506.159752:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617d500 - object 0x2c0000401:537 offset 0<>12288 00000020:00010000:2.0:1788572506.189911:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3998 (ffff92fc6721a3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572506.189919:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3998 (ffff92fc6721a3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.189923:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550330 (ffff92fc6721abd8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572506.189932:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.189948:0:109827:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:2.0:1788572506.189965:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3998 (ffff92fc6721a3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572506.189968:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550330 (ffff92fc6721a3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572506.189976:0:109827:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000080:00200000:2.0:1788572506.189988:0:109827:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572506.189993:0:109827:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000080:00200000:2.0:1788572506.189996:0:109827:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200008:2.0:1788572506.190014:0:109827:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 9000, count: 4185304 00000080:00200000:2.0:1788572506.190024:0:109827:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 9000, bytes: 4185304 00000080:00200000:2.0:1788572506.190028:0:109827:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 0 00000020:00200000:2.0:1788572506.190045:0:109827:0:(cl_io.c:214:cl_io_rw_init()) header@00000000204bb37a[0x110, 4, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572506.190047:0:109827:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [9000, 4194304) 0 0 00000080:00200000:2.0:1788572506.190051:0:109827:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1788572506.190067:0:109827:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 9000 chunk: [9000, 4194304) 4194304, 4185304 00020000:00200000:2.0:1788572506.190071:0:109827:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572506.190085:0:109827:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [2, 1023] 00000008:00010000:2.0:1788572506.190096:0:109827:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572506.190102:0:109827:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550330, osc lock ffff92fc58370ab0, flags 0x0 00010000:00010000:2.0:1788572506.190121:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190136:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (8192 4194303) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190148:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190158:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190166:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572506.190174:0:109827:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [9000, 4194304) 00000001:00010000:2.0:1788572506.190187:0:109827:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572506.190191:0:109827:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572506.190194:0:109827:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550b30, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572506.190199:0:109827:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190209:0:109827:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572506.190217:0:109827:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572506.190220:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190228:0:109827:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572506.190234:0:109827:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572506.190240:0:109827:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572506.190242:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572506.190245:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.190247:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550b30 (ffff92fc6721abd8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.190249:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70475280 (ffff92fc6721a3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572506.190252:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550b30 (ffff92fc6721a3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572506.190257:0:109827:0:(vvp_io.c:893:vvp_io_read_start()) count:4185304 iocb pos:9000 00000020:00010000:2.0:1788572506.190262:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3998 (ffff92fc6721a3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572506.190265:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3998 (ffff92fc6721a3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572506.190267:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550330 (ffff92fc6721abd8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572506.190271:0:109827:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572506.190280:0:109827:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc53e08000 lock: 00000000aeb69306/0x58a604dcef352440 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792379 expref: -99 pid: 109827 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:2.0:1788572506.190288:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3998 (ffff92fc6721a3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572506.190290:0:109827:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550330 (ffff92fc6721a3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572506.190293:0:109827:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:2.0:1788572506.190297:0:109827:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572506.190300:0:109827:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:2.0:1788572506.190302:0:109827:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:2.0:1788572506.192033:0:109827:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665e42c8), flags 100000 00000002:00100000:2.0:1788572506.192119:0:109827:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617e300 x1875449863845632/t4294983683(4294983683) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 592/696 e 0 to 0 dl 1788572522 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cmp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572506.192134:0:109827:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc53e08000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572506.192141:0:109827:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c000 x1875449863846144/t0(0) o35->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572506.192164:0:109827:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863846144:192.168.202.132@tcp:35:cmp.0 00000100:00100000:2.0:1788572506.192255:0:109827:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c87000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572506.200646:0:109827:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job cmp:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109827:1875449863846144:192.168.202.132@tcp:35:cmp.0 00000002:00100000:2.0:1788572506.200687:0:109827:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617e300 x1875449863845632/t4294983683(4294983683) o101->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 592/696 e 0 to 0 dl 1788572522 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cmp.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572506.835421:0:109845:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572506.835436:0:109845:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e2a08),name=/ 00000002:00010000:2.0:1788572506.835461:0:109845:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572506.835480:0:109845:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.835497:0:109845:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572506.835505:0:109845:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572506.835513:0:109845:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572506.835517:0:109845:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) for lock 0x58a604dcef35242b 00000080:00010000:2.0:1788572506.835524:0:109845:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35242b from it ffff9e038331fcb8 00010000:00010000:2.0:1788572506.835529:0:109845:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572506.835538:0:109845:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788572509.259770:0:109853:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572509.259785:0:109853:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e2a08),name=/ 00000002:00010000:2.0:1788572509.259812:0:109853:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572509.259836:0:109853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572509.259853:0:109853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572509.259863:0:109853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572509.259872:0:109853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572509.259876:0:109853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) for lock 0x58a604dcef35242b 00000080:00010000:2.0:1788572509.259886:0:109853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35242b from it ffff9e0383013cb8 00010000:00010000:2.0:1788572509.259890:0:109853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572509.259901:0:109853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200008:2.0:1788572509.260051:0:109853:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572509.260056:0:109853:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e2a08),name=/ 00000002:00010000:2.0:1788572509.260063:0:109853:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572509.260070:0:109853:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572509.260078:0:109853:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572509.260085:0:109853:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572509.260089:0:109853:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) 00000080:00010000:2.0:1788572509.260091:0:109853:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e2a08) for lock 0x58a604dcef35242b 00000080:00010000:2.0:1788572509.260095:0:109853:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35242b from it ffff9e0383013cb8 00010000:00010000:2.0:1788572509.260098:0:109853:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572509.260109:0:109853:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572509.299268:0:31488:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 1 restore needed 0 00020000:00200000:2.0:1788572509.299289:0:31488:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572509.299339:0:31488:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788572509.318198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863846656:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572509.325973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46155000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572509.325994:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863846656:192.168.202.132@tcp:9:umount.0 00000020:00000080:2.0:1788572509.328338:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990c000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc53e08000 00000020:00000080:2.0:1788572509.329249:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc46155000 for lustre-OST0001-osc-ffff92fc53e08000 00000020:00000080:2.0:1788572509.329257:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc46151000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 00000020:00000080:2.0:1788572509.329261:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46151000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 for lustre-OST0001-osc-ffff92fc53e08000 00000020:01000000:2.0:1788572509.329266:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc53e08000 (fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000100:00100000:1.0:1788572509.329629:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863846912:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572509.333680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46150800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572509.333716:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863846912:192.168.202.132@tcp:9:umount.0 00000020:00000080:2.0:1788572509.334774:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc49909000/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc53e08000 00000020:00000080:2.0:1788572509.338438:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc46150800 for lustre-OST0000-osc-ffff92fc53e08000 00000020:00000080:2.0:1788572509.338459:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc46153000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 00000020:00000080:2.0:1788572509.338463:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46153000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 for lustre-OST0000-osc-ffff92fc53e08000 00000020:01000000:2.0:1788572509.338472:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc53e08000 (fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000100:00080000:1.0:1788572509.339255:0:109853:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import fdebee5e-f7b1-4f7d-9968-d622b2cde851->lustre-MDT0000_UUID 00010000:00010000:1.0:1788572509.339411:0:109853:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572509.339475:0:109853:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572509.339482:0:109853:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572509.339500:0:109853:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce5a80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572509.339513:0:109853:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job umount:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109853:1875449863847040:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572509.339523:0:109853:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572509.339531:0:109853:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572509.339535:0:109853:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc702f5800/0x58a604dcef35242b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179232c expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572509.339864:0:109853:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572509.339884:0:109853:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572509.339889:0:109853:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572509.339895:0:109853:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce5a80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572509.339900:0:109853:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job umount:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109853:1875449863847168:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572509.339906:0:109853:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572509.339909:0:109853:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572509.339912:0:109853:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70366000/0x58a604dcef352432 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792348 expref: -99 pid: 109827 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572509.339941:0:109853:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572509.339957:0:109853:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572509.339963:0:109853:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572509.339967:0:109853:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce5a80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572509.339971:0:109853:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job umount:fdebee5e-f7b1-4f7d-9968-d622b2cde851:109853:1875449863847296:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572509.339976:0:109853:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572509.339979:0:109853:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572509.339982:0:109853:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc53e08000 lock: ffff92fc70367a00/0x58a604dcef352439 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792364 expref: -99 pid: 109827 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000100:00080000:1.0:1788572509.340778:0:109853:0:(import.c:65:import_set_state_nolock()) ffff92fc46157800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:2.0:1788572509.343636:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990f800/fdebee5e-f7b1-4f7d-9968-d622b2cde851 for lustre-clilov-ffff92fc53e08000 00000100:00080000:1.0:1788572509.355633:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46157800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572509.355671:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863847424:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572509.355777:0:109853:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572509.355782:0:109853:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc53e08000: committing for last_committed 4294983683 gen 2 00000100:00100000:2.0:1788572509.355793:0:109853:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67289c00 x1875449863847424/t4294983685(4294983685) o39->lustre-MDT0000-mdc-ffff92fc53e08000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572519 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572509.355818:0:109853:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572509.359908:0:109853:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352408 00000020:00000080:2.0:1788572509.359924:0:109853:0:(genops.c:885:class_export_put()) final put ffff92fc46155800/fdebee5e-f7b1-4f7d-9968-d622b2cde851 00000100:00100000:1.0:1788572509.364347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863847552:192.168.202.132@tcp:39:umount.0 00000100:00100000:1.0:1788572509.375797:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc53e08000: committing for last_committed 4294976746 gen 1 00000100:00080000:1.0:1788572509.375815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46151800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572509.375852:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fdebee5e-f7b1-4f7d-9968-d622b2cde851:2392:1875449863847552:192.168.202.132@tcp:39:umount.0 00000020:00000080:2.0:1788572509.379421:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46152800/fdebee5e-f7b1-4f7d-9968-d622b2cde851 for lustre-MDT0001-mdc-ffff92fc53e08000 00000020:00000080:2.0:1788572509.379477:0:109853:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352401 00000020:00000080:2.0:1788572509.379482:0:109853:0:(genops.c:885:class_export_put()) final put ffff92fc46153800/fdebee5e-f7b1-4f7d-9968-d622b2cde851 00000020:01000000:2.0:1788572509.379509:0:109853:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc53e08000 (flags='') 00000020:00000080:2.0:1788572509.379521:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572509.379528:0:109853:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc53e08000: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572509.379533:0:109853:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc48ccb380) has no exports 00000020:00000080:2.0:1788572509.379616:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572509.379623:0:109853:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc53e08000 (uuid fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000020:00000080:2.0:1788572509.379626:0:109853:0:(genops.c:885:class_export_put()) final put ffff92fc5106f000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 00000020:00000080:2.0:1788572509.379629:0:109853:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5106f000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 for lustre-clilov-ffff92fc53e08000 00000020:01000000:2.0:1788572509.379635:0:109853:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc53e08000 (fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000020:01000000:2.0:1788572509.379861:0:109853:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc53e08000 (flags='') 00000020:00000080:2.0:1788572509.379865:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572509.379868:0:109853:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc53e08000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572509.379871:0:109853:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc48cccd40) has no exports 00000020:00000080:2.0:1788572509.379922:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572509.379926:0:109853:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc53e08000 (uuid fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000020:01000000:2.0:1788572509.379930:0:109853:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc53e08000 (flags='') 00000020:00000080:2.0:1788572509.379932:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572509.379935:0:109853:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc53e08000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572509.379937:0:109853:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc48cc8000) has no exports 00000008:01000000:2.0:1788572509.380491:0:109853:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc53e08000: client import never connected 00000100:00080000:2.0:1788572509.380496:0:109853:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572509.380501:0:109853:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc53e08000: committing for last_committed 4294983683 gen 3 00000100:00100000:2.0:1788572509.380509:0:109853:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572509.384428:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572509.384440:0:109853:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc53e08000 (uuid fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000020:01000000:2.0:1788572509.384445:0:109853:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc53e08000 (flags='') 00000020:00000080:2.0:1788572509.384450:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572509.384454:0:109853:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc53e08000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572509.384458:0:109853:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc70033380) has no exports 00000008:01000000:1.0:1788572509.393835:0:109853:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc53e08000: client import never connected 00000100:00080000:1.0:1788572509.393845:0:109853:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572509.393850:0:109853:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc53e08000: committing for last_committed 4294976746 gen 3 00000100:00100000:1.0:1788572509.393860:0:109853:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572509.398266:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572509.398280:0:109853:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc53e08000 (uuid fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000080:02000400:1.0:1788572509.398286:0:109853:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572509.398288:0:109853:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:00000080:2.0:1788572509.401537:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc46151800 for lustre-MDT0001-mdc-ffff92fc53e08000 00000020:01000004:1.0:1788572509.404658:0:109853:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc53e08000 00000100:00080000:1.0:1788572509.404675:0:109853:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 4bf8bb21-fd31-446f-ac9d-6e5b1f746639->MGS 00010000:00010000:1.0:1788572509.404698:0:109853:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572509.404716:0:109853:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572509.404765:0:109853:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572509.404768:0:109853:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572509.404771:0:109853:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572509.404837:0:109853:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572509.404845:0:109853:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572509.404867:0:109853:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58ce5a80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572509.404881:0:109853:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job umount:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:109853:1875449863847680:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572509.404895:0:109853:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572509.404905:0:109853:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572509.404909:0:109853:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef3523bb lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217922ed expref: -99 pid: 109785 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572509.404944:0:109853:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572509.404972:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:2392:1875449863847808:192.168.202.132@tcp:251:umount.0 00000020:00000080:2.0:1788572509.405419:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc46152000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 00000020:00000080:2.0:1788572509.405425:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46152000/fdebee5e-f7b1-4f7d-9968-d622b2cde851 for lustre-MDT0001-mdc-ffff92fc53e08000 00000020:01000000:2.0:1788572509.405434:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc53e08000 (fdebee5e-f7b1-4f7d-9968-d622b2cde851) 00000100:00080000:0.0:1788572509.412212:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788572509.412214:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572509.412233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4bf8bb21-fd31-446f-ac9d-6e5b1f746639:2392:1875449863847808:192.168.202.132@tcp:251:umount.0 00000100:00080000:1.0:1788572509.412386:0:109853:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572509.412391:0:109853:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572509.412393:0:109853:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572509.412395:0:109853:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572509.412406:0:109853:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3523b4 00000020:00000080:1.0:1788572509.412416:0:109853:0:(genops.c:885:class_export_put()) final put ffff92fc58248800/4bf8bb21-fd31-446f-ac9d-6e5b1f746639 00000020:01000000:1.0:1788572509.412440:0:109853:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:1.0:1788572509.412447:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572509.412455:0:109853:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572509.412460:0:109853:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc48cc99c0) has no exports 10000000:01000000:1.0:1788572509.412506:0:109799:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:1.0:1788572509.412658:0:109853:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:1.0:1788572509.412661:0:109853:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572509.412663:0:109853:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572509.412665:0:109853:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572509.412667:0:109853:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572509.412757:0:109853:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572509.412765:0:109853:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 4bf8bb21-fd31-446f-ac9d-6e5b1f746639) 00000020:01000004:1.0:1788572509.412771:0:109853:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc53e08000 1 00000020:01000004:1.0:1788572509.412775:0:109853:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc4a3b0800 00000020:00000080:0.0:1788572509.413019:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58248800/4bf8bb21-fd31-446f-ac9d-6e5b1f746639 for MGC192.168.202.132@tcp 00000020:00000080:0.0:1788572509.413034:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5824c000 for MGC192.168.202.132@tcp 00000020:00000080:0.0:1788572509.413041:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc6090c800/4bf8bb21-fd31-446f-ac9d-6e5b1f746639 00000020:00000080:0.0:1788572509.413043:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6090c800/4bf8bb21-fd31-446f-ac9d-6e5b1f746639 for MGC192.168.202.132@tcp 00000020:01000000:0.0:1788572509.413048:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (4bf8bb21-fd31-446f-ac9d-6e5b1f746639) 00010000:00010000:0.0:1788572509.500170:0:109797:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00000100:00100000:0.0:1788572509.504885:0:109786:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc60ab3600:109786] 0 exiting: rc = 1 00000100:00100000:0.0:1788572509.506862:0:109787:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc49ba8400:109787] 1 exiting: rc = 0 00000100:00100000:2.0:1788572509.507511:0:109788:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc49ba8200:109788] 0 exiting: rc = 1 00000100:00100000:0.0:1788572509.507883:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.507894:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.508488:0:109790:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc470f3600:109790] 0 exiting: rc = 1 00000100:00100000:0.0:1788572509.508546:0:109791:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc49ba8600:109791] 1 exiting: rc = 0 00000100:00100000:1.0:1788572509.508965:0:109792:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc49ba9a00:109792] 2 exiting: rc = 0 00000100:00100000:2.0:1788572509.509140:0:109793:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc51c14400:109793] 0 exiting: rc = 1 00000100:00100000:0.0:1788572509.513811:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513822:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513829:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513835:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513841:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513846:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513853:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513858:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513863:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513868:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513874:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513879:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513884:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513889:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513895:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513901:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513906:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513912:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513918:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513924:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513930:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513936:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513942:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513948:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513955:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513960:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513967:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513972:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513978:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513984:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513989:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.513995:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514001:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514007:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514013:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514019:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514025:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514030:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514037:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514044:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514050:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514056:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514061:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514067:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514072:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514078:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514083:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514088:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514094:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514099:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514105:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514109:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514115:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514121:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514126:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514131:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514136:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514142:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514147:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514153:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514158:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514164:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514169:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514175:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514182:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514187:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514193:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514198:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514204:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514208:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514214:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514219:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514224:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514229:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514235:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514241:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514247:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514252:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514258:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514264:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514270:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514275:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514282:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514288:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514294:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514300:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514307:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514313:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514319:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514325:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514343:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514349:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514355:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514361:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514367:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514373:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514379:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514384:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514389:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514394:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514399:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514404:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514410:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514415:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514421:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514426:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514431:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514438:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514444:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514450:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514455:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514460:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514466:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514472:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514478:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514484:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514489:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514495:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514501:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514506:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514512:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514518:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514524:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514529:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514534:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514539:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514545:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572509.514552:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572509.727377:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572509.727393:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572509.740608:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:2.0:1788572509.740632:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:2.0:1788572509.748798:0:109866:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:2.0:1788572509.749123:0:109866:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:2.0:1788572509.749317:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:2.0:1788572509.749323:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788572509.750184:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788572509.750191:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:2.0:1788572509.750683:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:2.0:1788572509.750690:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:0.0:1788572509.751320:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:0.0:1788572509.751342:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:0.0:1788572509.752695:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:0.0:1788572509.752705:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:0.0:1788572509.753234:0:109866:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:0.0:1788572509.753242:0:109866:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:1.0:1788572509.754528:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572509.754531:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572509.754533:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:0.0:1788572509.755039:0:109866:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc48719000@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572509.755218:0:109866:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:0.0:1788572509.755225:0:109866:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:1.0:1788572509.768990:0:109881:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:2.0:1788572509.769082:0:109866:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid e7bf4cc7-6786-4e88-b978-fb531985db08) 00000020:00000080:2.0:1788572509.769113:0:109866:0:(genops.c:1385:class_connect()) connect: client e7bf4cc7-6786-4e88-b978-fb531985db08, cookie 0x58a604dcef35244e 00000100:00080000:2.0:1788572509.769143:0:109866:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572509.769156:0:109866:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc48719000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572509.769192:0:109866:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449863847936/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572509.769224:0:109866:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import e7bf4cc7-6786-4e88-b978-fb531985db08->MGS 00000080:00200000:2.0:1788572509.769246:0:109866:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc69d47000 (sb ffff92fc69d47000) 00000080:01000000:2.0:1788572509.769371:0:109866:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:2.0:1788572509.769373:0:109866:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:2.0:1788572509.769377:0:109866:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:2.0:1788572509.769385:0:109866:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 22e19d3b-9675-4547-8eb5-08aa5a0df26e 10000000:01000000:2.0:1788572509.769974:0:109866:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:2.0:1788572509.769979:0:109866:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc69d47000 10000000:01000000:2.0:1788572509.769983:0:109866:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc51cd99c0 10000000:01000000:2.0:1788572509.769988:0:109866:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572509.769995:0:109866:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc51cd99c0 from 1 10000000:01000000:2.0:1788572509.769998:0:109866:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788572509.770032:0:109866:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.770043:0:109866:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.770053:0:109866:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572509.770074:0:109866:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc4617e300 x1875449863848064/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572509.770085:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6097d540 going to sleep for 0 seconds 00000100:00100000:1.0:1788572509.770283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e7bf4cc7-6786-4e88-b978-fb531985db08:2392:1875449863847936:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788572509.798928:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572509.798950:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:0.0:1788572509.798960:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:0.0:1788572509.798966:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:0.0:1788572509.798971:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48719000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:0.0:1788572509.798974:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788572509.798985:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:0.0:1788572509.798997:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000839ef33c): req@ffff92fc4617e300 x1875449863848064/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572509.799021:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e7bf4cc7-6786-4e88-b978-fb531985db08:2392:1875449863847936:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572509.799424:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6097d540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572509.818381:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848064:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572509.818417:0:109866:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572509.818431:0:109866:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572509.818805:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848192:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572509.818906:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.824916:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848192:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:2.0:1788572509.824966:0:109866:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572509.824991:0:109866:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.825003:0:109866:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572509.825018:0:109866:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc69d47000 10000000:01000000:2.0:1788572509.825024:0:109866:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:2.0:1788572509.825029:0:109866:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc69d47000 10000000:01000000:2.0:1788572509.825034:0:109866:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572509.825038:0:109866:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc69d47000 10000000:01000000:2.0:1788572509.825040:0:109866:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:2.0:1788572509.825044:0:109866:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc69d47000 from 1 10000000:01000000:2.0:1788572509.825048:0:109866:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788572509.825270:0:109866:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.825279:0:109866:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.825290:0:109866:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572509.825313:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848320:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572509.825440:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fe40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.842015:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848320:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572509.842046:0:109866:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.842061:0:109866:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572509.842324:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848448:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572509.842453:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fe40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.846479:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848448:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572509.846565:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848576:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572509.846671:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fe40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.850185:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863848576:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572509.851096:0:109885:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:e7bf4cc7-6786-4e88-b978-fb531985db08:109885:1875449863848704:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572509.851190:0:109885:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.854552:0:109885:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:e7bf4cc7-6786-4e88-b978-fb531985db08:109885:1875449863848704:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572509.854605:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.854618:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.854620:0:109885:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1788572509.854627:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc69d47000 00000020:00000080:2.0:1788572509.854630:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572509.854643:0:109885:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc69d47000 (ffff92fc48d13380) 00000020:00000080:2.0:1788572509.854660:0:109885:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:2.0:1788572509.854667:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc69d47000 00000020:00000080:2.0:1788572509.854670:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572509.854979:0:109885:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:2.0:1788572509.854984:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.854987:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.854989:0:109885:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1788572509.854993:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.854996:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.854997:0:109885:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572509.855001:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc69d47000 00000020:00000080:2.0:1788572509.855003:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572509.855010:0:109885:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc69d47000 (ffff92fc48d14d40) 00000020:00000080:2.0:1788572509.855019:0:109885:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:2.0:1788572509.855023:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc69d47000 00000020:00000080:2.0:1788572509.855026:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572509.855195:0:109885:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:2.0:1788572509.855199:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.855202:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.855204:0:109885:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572509.855206:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.855209:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.855210:0:109885:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1788572509.855213:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1788572509.855215:0:109885:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1788572509.855218:0:109885:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1788572509.855223:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.855225:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.855226:0:109885:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1788572509.855228:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.855232:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.855233:0:109885:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1788572509.855236:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572509.855239:0:109885:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572509.855261:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.855263:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572509.855271:0:109885:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc69d47000 (ffff92fc48d119c0) 00000020:00000080:2.0:1788572509.855279:0:109885:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:2.0:1788572509.855283:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.855285:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572509.855362:0:109885:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5df7b000@lustre-MDT0000-mdc-ffff92fc69d47000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572509.857038:0:109885:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc69d47000-1 is initialized 00000040:01000000:2.0:1788572509.857272:0:109885:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc69d47000 ctxt 13 is initialized 00000020:00000080:2.0:1788572509.874978:0:109885:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:2.0:1788572509.875001:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc69d47000 00000020:00000080:2.0:1788572509.875006:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572509.875030:0:109885:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788572509.875129:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.875134:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.875135:0:109885:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788572509.875139:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572509.875142:0:109885:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:2.0:1788572509.875143:0:109885:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572509.875145:0:109885:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572509.875146:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572509.875149:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.875151:0:109885:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572509.875154:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.875425:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.875428:0:109885:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:2.0:1788572509.875436:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572509.875441:0:109885:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572509.875470:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.875473:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572509.875485:0:109885:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc69d47000 (ffff92fc48d10000) 00000020:00000080:2.0:1788572509.875502:0:109885:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:2.0:1788572509.875508:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.875512:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572509.875580:0:109885:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5df78000@lustre-MDT0001-mdc-ffff92fc69d47000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572509.878676:0:109885:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc69d47000-2 is initialized 00000040:01000000:2.0:1788572509.879062:0:109885:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc69d47000 ctxt 13 is initialized 00000020:00000080:2.0:1788572509.883896:0:109885:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:2.0:1788572509.883915:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc69d47000 00000020:00000080:2.0:1788572509.883920:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572509.883947:0:109885:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:2.0:1788572509.883981:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.883986:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.883988:0:109885:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:2.0:1788572509.883991:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572509.883994:0:109885:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:2.0:1788572509.883996:0:109885:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572509.883998:0:109885:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572509.884000:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572509.884002:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.884004:0:109885:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572509.884008:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572509.884015:0:109885:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572509.884019:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.884021:0:109885:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788572509.884025:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572509.884031:0:109885:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572509.884061:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.884065:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572509.884091:0:109885:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc69d47000 (ffff92fc48cc99c0) 00000020:00000080:2.0:1788572509.884109:0:109885:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788572509.884115:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.884118:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572509.884198:0:109885:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5df7d000@lustre-OST0000-osc-ffff92fc69d47000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572509.885802:0:109885:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc69d47000-3 is initialized 00000020:00000080:2.0:1788572509.886418:0:109885:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:2.0:1788572509.886432:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc69d47000 00000020:00000080:2.0:1788572509.886435:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572509.886457:0:109885:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788572509.886464:0:109885:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc69d47000: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788572509.886469:0:109885:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788572509.886483:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.886487:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.886489:0:109885:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788572509.886493:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572509.886496:0:109885:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:2.0:1788572509.886497:0:109885:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572509.886499:0:109885:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572509.886500:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572509.886503:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.886504:0:109885:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572509.886507:0:109885:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572509.886511:0:109885:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572509.886515:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.886516:0:109885:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788572509.886519:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572509.886524:0:109885:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572509.886546:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.886549:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572509.886559:0:109885:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc69d47000 (ffff92fc48cc8000) 00000020:00000080:2.0:1788572509.886574:0:109885:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:2.0:1788572509.886580:0:109885:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc69d47000 00000020:00000080:2.0:1788572509.886582:0:109885:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572509.886650:0:109885:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc6714d000@lustre-OST0001-osc-ffff92fc69d47000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572509.887559:0:109885:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc69d47000-4 is initialized 00000020:01000000:0.0:1788572509.908875:0:109866:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:0.0:1788572509.908901:0:109866:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572509.908919:0:109866:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572509.908933:0:109866:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572509.908941:0:109866:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc69d47000 from 1 10000000:01000000:0.0:1788572509.908944:0:109866:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572509.909055:0:109866:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572509.909064:0:109866:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572509.909075:0:109866:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572509.909100:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849088:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572509.909185:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fe40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.915502:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849088:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572509.915536:0:109866:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.915551:0:109866:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572509.922030:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849216:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572509.922148:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.941681:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849216:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572509.942428:0:109866:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572509.942456:0:109866:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.942468:0:109866:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572509.942477:0:109866:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc69d47000 from 1 10000000:01000000:2.0:1788572509.942480:0:109866:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572509.942521:0:109866:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.942529:0:109866:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.942539:0:109866:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572509.942563:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849344:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572509.942638:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.950265:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849344:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572509.950307:0:109866:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572509.950322:0:109866:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572509.950433:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849472:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572509.950520:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572509.954994:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849472:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572509.955065:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849600:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572509.955163:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc583290c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.961796:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:e7bf4cc7-6786-4e88-b978-fb531985db08:109866:1875449863849600:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572509.962924:0:109891:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:e7bf4cc7-6786-4e88-b978-fb531985db08:109891:1875449863849728:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572509.963043:0:109891:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.969246:0:109891:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:e7bf4cc7-6786-4e88-b978-fb531985db08:109891:1875449863849728:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572509.969297:0:109891:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.969311:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.969314:0:109891:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572509.969322:0:109891:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc69d47000 00000020:00000080:2.0:1788572509.969334:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572509.969343:0:109891:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.969345:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.969347:0:109891:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572509.969349:0:109891:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572509.969352:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.969353:0:109891:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572509.969356:0:109891:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc69d47000 00000020:00000080:2.0:1788572509.969358:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572509.969940:0:109891:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572509.978547:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572509.978557:0:109891:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:1.0:1788572509.978570:0:109891:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572509.978703:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572509.978705:0:109891:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:1.0:1788572509.978712:0:109891:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc69d47000 00000020:00000080:1.0:1788572509.978715:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:1.0:1788572509.979476:0:109891:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 721us 00000100:00100000:1.0:1788572509.979546:0:109891:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:e7bf4cc7-6786-4e88-b978-fb531985db08:109891:1875449863849856:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572509.980677:0:109891:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572509.983032:0:109891:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:e7bf4cc7-6786-4e88-b978-fb531985db08:109891:1875449863849856:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572509.983079:0:109891:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572509.983088:0:109891:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572509.983091:0:109891:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:2.0:1788572509.983095:0:109891:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:0.0:1788572509.984439:0:109866:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:0.0:1788572509.984467:0:109866:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572509.984496:0:109866:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572509.984510:0:109866:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:0.0:1788572509.984537:0:109866:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572509.984561:0:109866:0:(genops.c:1385:class_connect()) connect: client 22e19d3b-9675-4547-8eb5-08aa5a0df26e, cookie 0x58a604dcef35249b 00800000:01000000:0.0:1788572509.984585:0:109866:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 22e19d3b-9675-4547-8eb5-08aa5a0df26e to lustre-clilmv-ffff92fc69d47000 00800000:01000000:0.0:1788572509.984593:0:109866:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc69d47000(22e19d3b-9675-4547-8eb5-08aa5a0df26e) - lustre-MDT0000_UUID, 22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:00000080:0.0:1788572509.984601:0:109866:0:(genops.c:1385:class_connect()) connect: client 22e19d3b-9675-4547-8eb5-08aa5a0df26e, cookie 0x58a604dcef3524a2 00000100:00080000:0.0:1788572509.984612:0:109866:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc69d47000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572509.984623:0:109866:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc69d47000: import ffff92fc5df7b000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572509.984636:0:109866:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572509.984665:0:109866:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863849984/t0(0) o38->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572509.984694:0:109866:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-MDT0000_UUID 00000100:00100000:2.0:1788572509.984817:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863849984:192.168.202.132@tcp:38:mount.lustre.0 00800000:01000000:0.0:1788572509.985042:0:109866:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc69d47000(22e19d3b-9675-4547-8eb5-08aa5a0df26e) successfully (5) 00800000:00000004:0.0:1788572509.985047:0:109866:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc69d47000: age 3265, max_age 0 00800000:01000000:0.0:1788572509.985096:0:109866:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc69d47000(22e19d3b-9675-4547-8eb5-08aa5a0df26e) - lustre-MDT0001_UUID, 22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:00000080:0.0:1788572509.985104:0:109866:0:(genops.c:1385:class_connect()) connect: client 22e19d3b-9675-4547-8eb5-08aa5a0df26e, cookie 0x58a604dcef3524a9 00000100:00080000:0.0:1788572509.985110:0:109866:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc69d47000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572509.985116:0:109866:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc69d47000: import ffff92fc5df78000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572509.985121:0:109866:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572509.985286:0:109866:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449863850240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572509.985310:0:109866:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-MDT0001_UUID 00000100:00100000:2.0:1788572509.985461:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863850240:192.168.202.132@tcp:38:mount.lustre.0 00000100:00080000:1.0:1788572509.986274:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704b480 x1875449863850112/t0(0) o41->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00800000:01000000:0.0:1788572509.990266:0:109866:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc69d47000(22e19d3b-9675-4547-8eb5-08aa5a0df26e) successfully (5) 00800000:00000004:0.0:1788572509.990273:0:109866:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc69d47000: age 3265, max_age 0 00000080:00000004:0.0:1788572509.990353:0:109866:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc69d47000: age 3265, max_age 4264 00000100:00080000:1.0:1788572509.990354:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704bb80 x1875449863850368/t0(0) o41->lustre-MDT0001-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00800000:00000004:0.0:1788572509.990359:0:109866:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc69d47000: age 3265, max_age 4264 00000100:00080000:0.0:1788572509.990376:0:109866:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67049180 x1875449863850496/t0(0) o41->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572509.990384:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788572510.006998:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc69d47000: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572510.007016:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc69d47000: connect to target with instance 1 00000100:00080000:2.0:1788572510.007027:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc69d47000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572510.007040:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:2.0:1788572510.007044:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5df78000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:1.0:1788572510.020199:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:0:1875449863850368:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572510.023643:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109866:1875449863850496:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:0.0:1788572510.023675:0:109866:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc69d47000: update ffff92fc48d11e18 cache blocks 317012/353529 objects 1023701/1024000 00000080:00000004:0.0:1788572510.023682:0:109866:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc69d47000: update ffff92fc48d15198 cache blocks 317012/353529 objects 1023701/1024000 00000100:00100000:0.0:1788572510.023757:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109866:1875449863850624:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:0.0:1788572510.023841:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fd80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572510.030785:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:0:1875449863850112:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572510.038765:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109866:1875449863850624:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:0.0:1788572510.038797:0:109866:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:0.0:1788572510.038812:0:109866:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:0.0:1788572510.038817:0:109866:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:0.0:1788572510.038829:0:109866:0:(genops.c:1385:class_connect()) connect: client 22e19d3b-9675-4547-8eb5-08aa5a0df26e, cookie 0x58a604dcef3524b0 00000020:00000080:0.0:1788572510.038846:0:109866:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef3524b7 00000100:00080000:0.0:1788572510.038854:0:109866:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc69d47000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572510.038864:0:109866:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc69d47000: import ffff92fc5df7d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572510.038876:0:109866:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572510.038902:0:109866:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449863850752/t0(0) o8->lustre-OST0000-osc-ffff92fc69d47000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572510.038936:0:109866:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-OST0000_UUID 00020000:01000000:0.0:1788572510.038962:0:109866:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc69d47000) active 00000020:00000080:0.0:1788572510.038990:0:109866:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef3524be 00000100:00080000:0.0:1788572510.038994:0:109866:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc69d47000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572510.038999:0:109866:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc69d47000: import ffff92fc6714d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572510.039005:0:109866:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572510.039017:0:109866:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449863850880/t0(0) o8->lustre-OST0001-osc-ffff92fc69d47000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572510.039027:0:109866:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-OST0001_UUID 00020000:01000000:0.0:1788572510.039033:0:109866:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc69d47000) active 00000100:00100000:0.0:1788572510.039071:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109866:1875449863851008:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572510.039139:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fd80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572510.051708:0:109866:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109866:1875449863851008:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572510.051814:0:109866:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109866:1875449863851136:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572510.051940:0:109866:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fd80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572510.571586:0:109922:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:2.0:1788572510.571631:0:109922:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc69d47000 00010000:00010000:2.0:1788572510.571655:0:109922:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc69d47000 00010000:00010000:2.0:1788572510.571679:0:109922:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc69d47000 00010000:00010000:2.0:1788572510.571705:0:109922:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc69d47000 00010000:00010000:2.0:1788572510.800667:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788572511.644237:0:109930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:1.0:1788572511.644282:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572511.644577:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.644590:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.644600:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572511.644640:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851264:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572511.644782:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572511.649342:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851264:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572511.649372:0:109930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649385:0:109930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649398:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649412:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649425:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649432:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649440:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572511.649452:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67288a80 x1875449863851264/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572511.649466:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649482:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572511.649489:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572511.649521:0:109930:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572511.649525:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:1.0:1788572511.649530:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:1.0:1788572511.649534:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e0383697bf8 00010000:00010000:1.0:1788572511.649537:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649545:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572511.649573:0:109930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572511.649596:0:109930:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665dc2c8), flags=25 00000080:00200000:1.0:1788572511.649601:0:109930:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665dc2c8), intent=0 00000002:00010000:1.0:1788572511.649633:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572511.649675:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649682:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.649691:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572511.649711:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851392:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572511.649815:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572511.664429:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851392:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572511.664458:0:109930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664486:0:109930:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664495:0:109930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664506:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664517:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664529:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664538:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664547:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572511.664559:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc67288a80 x1875449863851392/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572511.664572:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664582:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:1.0:1788572511.664590:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572511.664694:0:109930:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8) 00000080:00200000:1.0:1788572511.664704:0:109930:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572511.664764:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d42c8) for lock 0x58a604dcef3524cc 00000080:00010000:1.0:1788572511.664777:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d42c8) 00000080:00010000:1.0:1788572511.664782:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524cc from it ffff9e0383697ac8 00010000:00010000:1.0:1788572511.664785:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.664795:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572511.664889:0:109930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572511.664910:0:109930:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), file ffff92fc50c43680, open_flags 4d902, mode 0 opened 0 00000080:00200000:1.0:1788572511.664915:0:109930:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=open 00000100:00100000:1.0:1788572511.664994:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851520:192.168.202.132@tcp:900:lfs.0 00000100:00100000:1.0:1788572511.665289:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572511.673578:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851520:192.168.202.132@tcp:900:lfs.0 00000002:00010000:1.0:1788572511.673632:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 01154403 00010000:00010000:1.0:1788572511.673695:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.673705:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572511.673716:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572511.673763:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67288a80 x1875449863851776/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572511.673773:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572511.673792:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851776:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572511.673950:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572511.713419:0:109930:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc69d47000: committing for last_committed 4294983686 gen 1 00000100:00100000:1.0:1788572511.713440:0:109930:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc67288a80 x1875449863851776/t4294983686(4294983686) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/696 e 0 to 0 dl 1788572527 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572511.713455:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851776:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572511.713480:0:109930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713501:0:109930:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713509:0:109930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713520:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713529:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713543:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713555:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713562:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572511.713571:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc67288a80 x1875449863851776/t4294983686(4294983686) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/696 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572511.713583:0:109930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67288a80 x1875449863851776/t4294983686(4294983686) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/696 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572511.713595:0:109930:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572511.713608:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.713619:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 4 res 8589935645/2 (0 0) 00000080:00200000:1.0:1788572511.713625:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572511.713655:0:109930:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00200000:1.0:1788572511.713806:0:109930:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:2.0:1788572511.719017:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef3524d3 00000080:00010000:2.0:1788572511.719090:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00200008:2.0:1788572511.719120:0:109930:0:(file.c:1090:ll_file_open()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), flags 1154402 00000002:00100000:2.0:1788572511.719165:0:109930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67288a80 x1875449863851776/t4294983686(4294983686) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 592/696 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572511.719223:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524d3 from it ffff92fc6701eb40 00010000:00010000:2.0:1788572511.719231:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.719243:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572511.720090:0:109930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), inode mode 81a4 mask 46 00000080:00200080:2.0:1788572511.720142:0:109930:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=800466af arg=7ffdfb7010bc 00000080:00200000:2.0:1788572511.720237:0:109930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572511.720566:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572511.720591:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.720609:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572511.720619:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572511.720628:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572511.720631:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572511.720636:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e0383697bf8 00010000:00010000:2.0:1788572511.720640:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.720649:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572511.720661:0:109930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572511.720667:0:109930:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=24 00000080:00200000:2.0:1788572511.720672:0:109930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572511.720687:0:109930:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572511.720692:0:109930:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=. :VOLATILE:0000:42BE3182:fd=03, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), file ffff92fc48170c00, open_flags 2f9c1, mode 8180 opened 0 00000080:00200000:2.0:1788572511.720699:0:109930:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=. :VOLATILE:0000:42BE3182:fd=03, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=open|creat 00000100:00100000:2.0:1788572511.720815:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851904:192.168.202.132@tcp:700:lfs.0 00000100:00100000:2.0:1788572511.720914:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.725919:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863851904:192.168.202.132@tcp:700:lfs.0 00000002:00010000:2.0:1788572511.725960:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: . :VOLATILE:0000:42BE3182:fd=}03,[0x20000041f:0x1:0x0]) in obj [0x20000041d:0x1:0x0], intent: open|creat flags 0574702 00010000:00010000:2.0:1788572511.725983:0:109930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.726049:0:109930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.726076:0:109930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572511.726079:0:109930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5c00/0x58a604dcef3524cc lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217923e2 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.726407:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef3524da lrc: 3/0,1 mode: --/CW res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.726416:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef3524da lrc: 3/0,1 mode: --/CW res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572511.726426:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572511.726436:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c700 x1875449863852160/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 584/66432 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572511.726446:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef3524da lrc: 3/0,1 mode: --/CW res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572511.726467:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863852160:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572511.726609:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.748381:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863852160:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.748414:0:109930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef3524da lrc: 4/0,1 mode: --/CW res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792413 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572511.757979:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572511.758455:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881866240 00000100:00100000:1.0:1788572511.758465:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881866240 00000100:00100000:1.0:1788572511.758523:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:1.0:1788572511.758533:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67049880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:6530:x1875449881866240:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572511.758551:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef3524da lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792413 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572511.758629:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572511.758636:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67049880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:6530:x1875449881866240:12345-192.168.202.132@tcp:104: Request processed in 104us (667us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572511.758643:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:2.0:1788572511.786172:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863852416:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.786197:0:109930:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786211:0:109930:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786219:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786228:0:109930:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786237:0:109930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786244:0:109930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205c00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572511.786250:0:109930:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786271:0:109930:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786278:0:109930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786286:0:109930:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572511.786292:0:109930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.786302:0:109930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205c00/0x58a604dcef3524e1 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572511.786339:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff92fc67339c00 x1875449863852416/t4294983688(4294983688) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/744 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572511.786353:0:109930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67339c00 x1875449863852416/t4294983688(4294983688) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/744 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572511.786369:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572511.786468:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041f:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572511.786476:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572511.786517:0:109930:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041f:0x1:0x0], updating i_size 0 00000002:00100000:2.0:1788572511.786537:0:109930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67339c00 x1875449863852416/t4294983688(4294983688) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 656/744 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572511.786578:0:109930:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67339c00 x1875449863852416/t4294983688(4294983688) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 656/744 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572511.786587:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572511.786592:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617f480 x1875449863852800/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572511.786614:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863852800:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572511.786691:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.803942:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863852800:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572511.803977:0:109930:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67339c00 x1875449863852416/t4294983688(4294983688) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 656/744 e 0 to 0 dl 1788572527 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572511.804034:0:109930:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572511.804042:0:109930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572511.804835:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572511.804858:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.804875:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572511.804884:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572511.804891:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572511.804894:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef3524d3 00000080:00010000:2.0:1788572511.804901:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524d3 from it ffff9e0383697cd8 00010000:00010000:2.0:1788572511.804905:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.804915:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572511.804941:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572511.804952:0:109930:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572511.804970:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572511.804974:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550cb0, osc lock ffff92fc58370ab0, flags 0x400001002 00010000:00010000:2.0:1788572511.805516:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.805528:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572511.805578:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788572511.805596:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572511.805618:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863852928:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.809172:0:2395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.809197:0:2395:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.809212:0:2395:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.809229:0:2395:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572511.809243:0:2395:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572511.809367:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572511.809425:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.809435:0:2395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.809449:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.809458:0:2395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572511.809478:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863852928:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.817763:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572511.822431:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041f:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788572511.822439:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572511.822442:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.822447:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049d5b0 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.822451:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572511.822455:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049d5b0 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572511.822468:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1788572511.822491:0:109930:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=c00c66f3 arg=7ffdfb703204 00000002:00010000:2.0:1788572511.822517:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: open flags 01004040000001 00010000:00010000:2.0:1788572511.822566:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.822574:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572511.822584:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572511.822592:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617dc00 x1875449863853440/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572511.822602:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572511.822621:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863853440:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572511.822987:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.826920:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863853440:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.826949:0:109930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.826963:0:109930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 4/1,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.826976:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.826987:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.827000:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.827608:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.827618:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 4/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572511.827638:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff92fc4617dc00 x1875449863853440/t4294983690(4294983690) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572511.827651:0:109930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617dc00 x1875449863853440/t4294983690(4294983690) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572511.827669:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.827686:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 2 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572511.827695:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1788572511.827877:0:109930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617dc00 x1875449863853440/t4294983690(4294983690) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572527 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572511.827887:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef3524fd 00000080:00010000:2.0:1788572511.827894:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524fd from it ffff9e0383697cc0 00010000:00010000:2.0:1788572511.827898:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.827906:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1788572511.827962:0:109930:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572511.827969:0:109930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572511.828177:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572511.828188:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572511.828206:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572511.828213:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572511.828219:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572511.828222:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef3524d3 00000080:00010000:2.0:1788572511.828226:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524d3 from it ffff9e0383697cd8 00010000:00010000:2.0:1788572511.828230:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572511.828238:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572511.828375:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:2.0:1788572511.828386:0:109930:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572511.828403:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572511.828406:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550030, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572511.828422:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.828435:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572511.828447:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572511.828450:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.828463:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572511.828472:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572511.828481:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572511.828485:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572511.828488:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.828490:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550030 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.828494:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572511.828496:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550030 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572511.828507:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572511.828590:0:109930:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572511.828596:0:109930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572511.828619:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572511.828626:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572511.828635:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572511.828641:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572511.828645:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572511.828648:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef3524d3 00000080:00010000:2.0:1788572511.828651:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524d3 from it ffff9e0383697cd8 00010000:00010000:2.0:1788572511.828655:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572511.828663:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572511.828675:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:2.0:1788572511.828679:0:109930:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572511.828686:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572511.828688:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550030, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572511.828695:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572511.828705:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572511.828714:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572511.828918:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572511.828928:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:2.0:1788572511.828936:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572511.828942:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572511.828944:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572511.828947:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.828950:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550030 (ffff92fc6721ae18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.828953:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572511.828955:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550030 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572511.828961:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:2.0:1788572511.828972:0:109930:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=801066da arg=7ffdfb703100 00000080:00200000:2.0:1788572511.828979:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1788572511.828985:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572511.834493:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863853568:192.168.202.132@tcp:1:lfs.0 00000080:00200008:2.0:1788572511.834998:0:109930:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572511.835164:0:109930:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572511.835233:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572511.835250:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572511.835371:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572511.835377:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572511.835384:0:109930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572511.835387:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef3524d3 00000080:00010000:2.0:1788572511.835394:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524d3 from it ffff9e0383697cd8 00010000:00010000:2.0:1788572511.835396:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572511.835404:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4200/0x58a604dcef3524d3 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217923fe expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572511.835424:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 restore needed 0 00000001:00010000:2.0:1788572511.835431:0:109930:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572511.835654:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572511.835658:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550030, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572511.835673:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572511.835682:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:2.0:1788572511.835691:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572511.835694:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572511.835699:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:2.0:1788572511.835705:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572511.835714:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572511.835792:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572511.835794:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.835796:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550030 (ffff92fc6721a2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.835799:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572511.835801:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550030 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572511.835809:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572511.871172:0:109930:0:(file.c:5214:ll_file_seek()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], offset: 0, type: (null) 00000080:00200000:0.0:1788572511.871216:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788572511.871225:0:109930:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:0.0:1788572511.871249:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572511.871252:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2730, osc lock ffff92fc45932850, flags 0x400001002 00010000:00010000:0.0:1788572511.871281:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572511.871303:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00200000:0.0:1788572511.871329:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572511.871334:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572511.871344:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000020:00200000:0.0:1788572511.871359:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572511.871370:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:0.0:1788572511.871375:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbc6f0 (ffff92fc582551b8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572511.871379:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbc6f0 (ffff92fc582551b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.871383:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2730 (ffff92fc58255d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.871387:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbc6f0 (ffff92fc582551b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572511.871389:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2730 (ffff92fc582551b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572511.871404:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572511.871415:0:109930:0:(file.c:5245:ll_file_seek()) COMPLETED file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], offset: 0, type: (null), rc = 12288 00000080:00200008:0.0:1788572511.871420:0:109930:0:(file.c:5214:ll_file_seek()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_HOLE 00000080:00200000:0.0:1788572511.871425:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:0.0:1788572511.871431:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572511.871450:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000080:00200000:0.0:1788572511.871458:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572511.871463:0:109930:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 0 00000080:00200000:0.0:1788572511.871465:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572511.871469:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572511.871472:0:109930:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 18446744073709551615] 00000008:00010000:0.0:1788572511.871479:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:0.0:1788572511.871481:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2730, osc lock ffff92fc45932850, flags 0x0 00010000:00010000:0.0:1788572511.871489:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572511.871498:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572511.871507:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572511.871515:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572511.871523:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572511.871533:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000100:00100000:1.0:1788572511.871701:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863853696:192.168.202.132@tcp:23:lfs.0 00000100:00100000:1.0:1788572511.877883:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863853696:192.168.202.132@tcp:23:lfs.0 00000020:00010000:0.0:1788572511.877971:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572511.877976:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.877981:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2730 (ffff92fc58255d88/1) at cl_lock_cancel():142 00010000:00010000:0.0:1788572511.877991:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572511.878007:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000020:00010000:0.0:1788572511.878026:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572511.878029:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2730 (ffff92fc582551b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572511.878045:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572511.878067:0:109930:0:(file.c:5245:ll_file_seek()) COMPLETED file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_HOLE, rc = 12288 00000080:00200080:0.0:1788572511.878094:0:109930:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=66f4 arg=10ab 00000080:00200008:0.0:1788572511.878110:0:109930:0:(file.c:2483:do_file_read_iter()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], ppos: 0, count: 67108864 00000080:00200000:0.0:1788572511.878120:0:109930:0:(file.c:2090:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: read ppos: 0, bytes: 67108864 00000080:00200000:0.0:1788572511.878128:0:109930:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 0 00000020:00200000:0.0:1788572511.878134:0:109930:0:(cl_io.c:214:cl_io_rw_init()) header@00000000cb670240[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:0.0:1788572511.878137:0:109930:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 67108864) 2048 0 00000080:00200000:0.0:1788572511.878141:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00000080:00200000:0.0:1788572511.878154:0:109930:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:0.0:1788572511.878162:0:109930:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:0.0:1788572511.878165:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572511.878181:0:109930:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:0.0:1788572511.878194:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:0.0:1788572511.878198:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2730, osc lock ffff92fc45932850, flags 0x40000 00000080:00200000:0.0:1788572511.878204:0:109930:0:(vvp_io.c:835:vvp_io_read_start()) crhIBIW0-TeUOeG6ka7IzA: read [0, 67108864) 00000001:00010000:0.0:1788572511.878214:0:109930:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:0.0:1788572511.878218:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572511.878220:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2230, osc lock ffff92fc45932be0, flags 0x400001002 00010000:00010000:0.0:1788572511.878238:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572511.878250:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00000020:00200000:0.0:1788572511.879185:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572511.879189:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572511.879196:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00000020:00200000:0.0:1788572511.879205:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572511.879212:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:0.0:1788572511.879215:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbc6f0 (ffff92fc582551b8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572511.879217:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbc6f0 (ffff92fc582551b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.879220:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2230 (ffff92fc58255d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.879222:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbc6f0 (ffff92fc582551b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572511.879224:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2230 (ffff92fc582551b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572511.879231:0:109930:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:0.0:1788572511.879236:0:109930:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), size=67108864 (max 536870912), offset=0=0x0, pages 16384 (max 131072), lockless, parallel 00000020:00200000:0.0:1788572511.880214:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:0.0:1788572511.880218:0:109930:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 12288 count 12288 file_offset 12288 00000020:00010000:0.0:1788572511.880225:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572511.880228:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.880230:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2730 (ffff92fc58255d88/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572511.880232:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572511.880234:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2730 (ffff92fc582551b8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572511.880240:0:109930:0:(vvp_io.c:533:vvp_io_advance()) advancing 12288 bytes 00000020:00200000:0.0:1788572511.880247:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1788572511.880300:0:108280:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc4133bc00 00000008:00100000:0.0:1788572511.880465:0:108280:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc67049500 - object 0x2c0000401:537 offset 0<>12288 00000100:00100000:1.0:1788572511.880572:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863853824:192.168.202.132@tcp:3:lfs.0 00000008:00010000:1.0:1788572511.885638:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc4133bc00 00000020:00200000:1.0:1788572511.885650:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:1.0:1788572511.885653:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:1.0:1788572511.885654:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1788572511.885663:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788572511.885682:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc4133bc00 00000100:00100000:1.0:1788572511.885708:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863853824:192.168.202.132@tcp:3:lfs.0 00000080:00200000:0.0:1788572511.885979:0:109930:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:0.0:1788572511.885993:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572511.885999:0:109930:0:(file.c:2238:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: 1 io complete with rc: 0, result: 12288, restart: 0 00000020:00200000:0.0:1788572511.886002:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572511.886011:0:109930:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 12288 00000080:00200008:0.0:1788572511.886040:0:109930:0:(file.c:2650:do_file_write_iter()) START file . :VOLATILE:0000:42BE3182:fd=03:[0x20000041f:0x1:0x0], ppos: 0, count: 12288 00000080:00200000:0.0:1788572511.886046:0:109930:0:(file.c:2090:ll_file_io_generic()) . :VOLATILE:0000:42BE3182:fd=03: write ppos: 0, bytes: 12288 00000080:00200000:0.0:1788572511.886054:0:109930:0:(file.c:1848:ll_io_set_mirror()) . :VOLATILE:0000:42BE3182:fd=03: desiginated mirror: 0 00000020:00200000:0.0:1788572511.886133:0:109930:0:(cl_io.c:214:cl_io_rw_init()) header@00000000ddb15274[0x110, 2, [0x20000041f:0x1:0x0] hash] 00000020:00200000:0.0:1788572511.886136:0:109930:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 12288) 2048 0 00000080:00200000:0.0:1788572511.886141:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041f:0x1:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00000080:00200000:0.0:1788572511.886157:0:109930:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:0.0:1788572511.886165:0:109930:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 12288) 12288, 12288 00020000:00200000:0.0:1788572511.886168:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572511.886187:0:109930:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 2] 00000008:00010000:0.0:1788572511.886200:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:0.0:1788572511.886205:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2730, osc lock ffff92fc45932850, flags 0x40000 00000080:00200000:0.0:1788572511.886215:0:109930:0:(vvp_io.c:1292:vvp_io_write_start()) . :VOLATILE:0000:42BE3182:fd=03: write [0, 12288) append 0 append lockpos 0 00000002:00010000:0.0:1788572511.886250:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x20000041f:0x1:0x0], intent: getxattr flags 0 00010000:00010000:0.0:1788572511.886501:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 3/1,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572511.886591:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 3/1,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572511.886605:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 3/1,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572511.886628:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863853952:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572511.887154:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.891999:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863853952:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.892197:0:109930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 4/1,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.892219:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 4/1,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.892231:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 4/1,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572511.892247:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc6704a300 x1875449863853952/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 568/1152 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572511.892447:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 3/1,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.892462:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 4 res 8589935647/1 (0 0) 00000080:00200000:2.0:1788572511.892469:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x20000041f:0x1:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572511.892496:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041f:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef352504 00000080:00010000:2.0:1788572511.892504:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352504 from it ffff9e03836978e8 00010000:00010000:2.0:1788572511.892508:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 3/1,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.892591:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572511.892634:0:109930:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041f:0x1:0x0](ffff92fc665d9988), size=12288 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel 00000020:00200000:2.0:1788572511.892708:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:2.0:1788572511.892711:0:109930:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 12288 count 12288 file_offset 12288 00000080:00200000:2.0:1788572511.892760:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041f:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1788572511.892766:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572511.892770:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.892774:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2730 (ffff92fc58255d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572511.892777:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c6c8 (ffff92fc582551b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572511.892780:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2730 (ffff92fc582551b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572511.892788:0:109930:0:(vvp_io.c:533:vvp_io_advance()) advancing 12288 bytes 00000020:00200000:2.0:1788572511.892797:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572511.892886:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc451f50c0 00000008:00100000:2.0:1788572511.893126:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617f100 - object 0x280000401:552 offset 0<>12288 00000100:00100000:0.0:1788572511.898943:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572511.899058:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881866368 00000100:00100000:1.0:1788572511.899123:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881866368 00000100:00100000:1.0:1788572511.899217:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 1 00000100:00100000:1.0:1788572511.899228:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67048e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8449:x1875449881866368:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572511.899249:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000015f6443b/0x58a604dcef3524f6 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a22179244b expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572511.899683:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572511.899695:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67048e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8449:x1875449881866368:12345-192.168.202.132@tcp:104: Request processed in 466us (760us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572511.899705:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 1 00010000:00010000:1.0:1788572511.901743:0:109877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000015f6443b/0x58a604dcef3524f6 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a22179244b expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572511.901751:0:109877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572511.901788:0:109877:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000015f6443b/0x58a604dcef3524f6 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a22179244b expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572511.901797:0:109877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000015f6443b/0x58a604dcef3524f6 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a22179244b expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572511.901914:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863854208:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572511.902146:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572511.902150:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572511.902152:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572511.907333:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863854208:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572511.913763:0:2395:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc69d47000: committing for last_committed 4294968532 gen 1 00000008:00010000:2.0:1788572511.913806:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc451f50c0 00000020:00200000:2.0:1788572511.913873:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:2.0:1788572511.913877:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:2.0:1788572511.913878:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1788572511.913886:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572511.913900:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc451f50c0 00000100:00100000:2.0:1788572511.913924:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863854080:192.168.202.132@tcp:4:lfs.0 00000080:00200000:0.0:1788572511.914229:0:109930:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:0.0:1788572511.914244:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572511.914251:0:109930:0:(file.c:2238:ll_file_io_generic()) . :VOLATILE:0000:42BE3182:fd=03: 2 io complete with rc: 0, result: 12288, restart: 0 00000080:00200000:0.0:1788572511.914451:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041f:0x1:0x0] updating i_size 12288 i_blocks 24 00000020:00200000:0.0:1788572511.914455:0:109930:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572511.914465:0:109930:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 12288 00000080:00200008:0.0:1788572511.914494:0:109930:0:(file.c:5384:ll_fsync()) START file: name=. :VOLATILE:0000:42BE3182:fd=03, fid=[0x20000041f:0x1:0x0], start=0, end=9223372036854775807, datasync=0, need_sync 1/1 00000100:00100000:0.0:1788572511.914561:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863854336:192.168.202.132@tcp:44:lfs.0 00000100:00100000:0.0:1788572511.914654:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fb40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.921041:0:109930:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc69d47000: committing for last_committed 4294983690 gen 1 00000100:00100000:2.0:1788572511.921072:0:109930:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617c700 x1875449863852160/t4294983687(4294983687) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572527 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572511.921139:0:109930:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617dc00 x1875449863853440/t4294983690(4294983690) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572527 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572511.921152:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863854336:192.168.202.132@tcp:44:lfs.0 00000080:00200000:2.0:1788572511.921176:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041f:0x1:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00020000:00200000:2.0:1788572511.921189:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572511.921281:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572511.921297:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863854464:192.168.202.132@tcp:16:lfs.0 00000100:00100000:2.0:1788572511.926766:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863854464:192.168.202.132@tcp:16:lfs.0 00000080:00200000:0.0:1788572511.926967:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041f:0x1:0x0] updating i_size 12288 i_blocks 24 00000080:00200000:0.0:1788572511.926983:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200080:1.0:1788572511.948175:0:109930:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=801066da arg=7ffdfb703100 00000080:00200000:1.0:1788572511.948199:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572511.948211:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788572511.948451:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572511.948476:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863854592:192.168.202.132@tcp:1:lfs.0 00000100:00100000:1.0:1788572511.955776:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863854592:192.168.202.132@tcp:1:lfs.0 00000080:00200008:2.0:1788572511.956543:0:109930:0:(llite_lib.c:2427:ll_setattr_raw()) START file . :VOLATILE:0000:42BE3182:fd=03:[0x20000041f:0x1:0x0](ffff92fc665d9988) current size 12288, valid attrs 101f0, mode 92fc, uid 1173, gid 1, new size 4294967296, atime 1788572496.000000000, mtime 1788572501.000000000, ctime 1788572511.000000000 00010000:00010000:2.0:1788572511.956585:0:109930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204e00/0x58a604dcef3524ef lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x1a/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792444 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572511.956606:0:109930:0:(vvp_object.c:117:vvp_conf_set()) [0x20000041f:0x1:0x0]: losing layout lock 00010000:00010000:2.0:1788572511.956657:0:109930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204e00/0x58a604dcef3524ef lrc: 1/0,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792444 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.956664:0:109930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572511.956666:0:109930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204e00/0x58a604dcef3524ef lrc: 0/0,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792444 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572511.956687:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572511.956698:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617f480 x1875449863854848/t0(0) o36->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572511.956774:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863854848:192.168.202.132@tcp:36:lfs.0 00000100:00100000:2.0:1788572511.957572:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.962443:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863854848:192.168.202.132@tcp:36:lfs.0 00000080:00200000:2.0:1788572511.962530:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041f:0x1:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00010000:00010000:2.0:1788572511.962555:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 31 res 8589935647/1 (0 0) 00000080:00010000:2.0:1788572511.962566:0:109930:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000041f:0x1:0x0](ffff92fc665d9988) 00000002:00010000:2.0:1788572511.962576:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041f:0x1:0x0]) in obj [0x20000041f:0x1:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572511.962629:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: --/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.962641:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: --/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.962651:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: --/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572511.962670:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863854976:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572511.962776:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.966899:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863854976:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.966930:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 4/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.966944:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 4/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572511.966960:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff92fc4617f100 x1875449863854976/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 376/480 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572511.966971:0:109930:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572511.966983:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.966997:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 16 res 8589935647/1 (0 0) 00000080:00200000:2.0:1788572511.967003:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000041f:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572511.967030:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041f:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef35250b 00000080:00010000:2.0:1788572511.967040:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35250b from it ffff9e0383697a78 00010000:00010000:2.0:1788572511.967043:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.967054:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 2/0,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.967076:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 2/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.967087:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 2/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572511.967095:0:109930:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000041f:0x1:0x0](00000000280ec2b7) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572511.967131:0:109930:0:(file.c:6843:ll_layout_conf()) [0x20000041f:0x1:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:2.0:1788572511.967136:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 3/1,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.967144:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 2/0,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788572511.967168:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572511.967183:0:109930:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788572511.967196:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 8 00000008:00010000:2.0:1788572511.967200:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550030, osc lock ffff92fc58371690, flags 0x0 00010000:00010000:2.0:1788572511.967243:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.967589:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.967602:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 3/0,1 mode: --/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572511.967626:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863855104:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572511.967793:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572511.981579:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863855104:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572511.981615:0:109930:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.981637:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.981652:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572511.981665:0:109930:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 12288, setting rss=12288; kms=12288, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.981677:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572511.981691:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572511.987504:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863855232:192.168.202.132@tcp:10:lfs.0 00000080:00200000:1.0:1788572511.987651:0:109930:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041f:0x1:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:1.0:1788572511.987660:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f286c8 (ffff92fc49fce008/0) at cl_lock_release():245 00000020:00010000:1.0:1788572511.987664:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f286c8 (ffff92fc49fce008/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572511.987668:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550030 (ffff92fc49fcec68/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572511.987679:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572511.987693:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788572511.987714:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f286c8 (ffff92fc49fce008/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572511.987717:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550030 (ffff92fc49fce008/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572511.987927:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00800000:00000080:1.0:1788572511.987943:0:109930:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff92fc69d47000: cmd=401866d4 len=392 karg=ffff92fc4900ce00 uarg=0000000000000000 00000002:00000080:1.0:1788572511.987956:0:109930:0:(mdc_request.c:2317:mdc_iocontrol()) lustre-MDT0000-mdc-ffff92fc69d47000: cmd=401866d4 len=392 karg=ffff92fc4900ce00 uarg=0000000000000000 00000020:00100000:1.0:1788572511.987984:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 55, max 7 00000100:00100000:1.0:1788572511.987993:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67289c00 x1875449863855488/t0(0) o55->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 472/224 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572511.988017:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863855488:192.168.202.132@tcp:55:lfs.0 00000100:00100000:1.0:1788572511.988427:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572511.992038:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572511.992395:0:109868:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881866496 00000100:00100000:0.0:1788572511.992406:0:109868:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881866496 00000100:00100000:0.0:1788572511.992509:0:109868:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 2 00000100:00100000:0.0:1788572511.992600:0:109868:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6728b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:6529:x1875449881866496:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572511.992620:0:109868:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572511.992706:0:109868:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572511.992714:0:109868:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6728b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:6529:x1875449881866496:12345-192.168.202.132@tcp:104: Request processed in 118us (684us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572511.992744:0:109868:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 2 00010000:00010000:2.0:1788572511.992802:0:109878:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572511.992810:0:109878:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992829:0:109878:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3c00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572511.992834:0:109878:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992851:0:109878:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 2/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992860:0:109878:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 3/0,0 mode: PR/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992914:0:109878:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 2/0,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992921:0:109878:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572511.992954:0:109878:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 1/0,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992961:0:109878:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3c00/0x58a604dcef352504 lrc: 0/0,0 mode: --/PR res: [0x20000041f:0x1:0x0].0x0 bits 0x20/0x20 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a22179247c expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572511.992975:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572511.992977:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572511.992979:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572511.992987:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572511.993002:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863855616:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572512.010555:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863855616:192.168.202.132@tcp:103:ldlm_bl.0 00000002:00010000:2.0:1788572512.012638:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.012653:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 16 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572512.012660:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572512.012679:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef352519 00000080:00010000:2.0:1788572512.012687:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352519 from it ffff9e0383697a78 00010000:00010000:2.0:1788572512.012691:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.012700:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.013034:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.013048:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572512.013058:0:109930:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000041d:0x2:0x0](00000000e0e21c90) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572512.013377:0:109930:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 1 00010000:00010000:2.0:1788572512.013384:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.013391:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788572512.013412:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572512.013426:0:109930:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1788572512.013441:0:109930:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572512.013445:0:109930:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550fb0, osc lock ffff92fc58370be0, flags 0x0 00010000:00010000:2.0:1788572512.013457:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc70499800 type 11 mode 2 res 11811161089/537 (0 18446744073709551615) 00010000:00010000:2.0:1788572512.013492:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.013500:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.013585:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.013615:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863856128:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.013781:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572512.017677:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.018000:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881866624 00000100:00100000:0.0:1788572512.018010:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881866624 00000100:00100000:0.0:1788572512.018118:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 3 00000100:00100000:0.0:1788572512.018129:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6728aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:130910:x1875449881866624:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572512.018149:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572512.018296:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572512.018304:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6728aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:130910:x1875449881866624:12345-192.168.202.132@tcp:104: Request processed in 176us (636us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572512.018311:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 3 00010000:00010000:2.0:1788572512.018617:0:109877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572512.018626:0:109877:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572512.018640:0:109877:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367e00 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:2.0:1788572512.018645:0:109877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572512.018700:0:109877:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572512.018712:0:109877:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572512.018773:0:109877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572512.018781:0:109877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572512.018823:0:109877:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572512.018833:0:109877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000007569468/0x58a604dcef3524e8 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a221792436 expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572512.018857:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.018859:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.018861:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572512.027833:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863856128:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.027874:0:109930:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.027889:0:109930:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.027898:0:109930:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572512.030625:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.030973:0:109868:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881866752 00000100:00100000:0.0:1788572512.030983:0:109868:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881866752 00000100:00100000:0.0:1788572512.031586:0:109868:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 4 00000100:00100000:0.0:1788572512.031601:0:109868:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6728b800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6513:x1875449881866752:12345-192.168.202.132@tcp:105: 00010000:00010000:0.0:1788572512.031619:0:109868:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.031633:0:109868:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.031642:0:109868:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.031652:0:109868:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.031664:0:109868:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10080000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.031674:0:109868:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x80000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.031705:0:109868:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff92fc70366c00) 00010000:00010000:2.0:1788572512.031809:0:109930:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.031830:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.031853:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572512.031865:0:109930:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.031877:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.031885:0:109930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572512.032622:0:109868:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6728b800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:6513:x1875449881866752:12345-192.168.202.132@tcp:105: Request processed in 1023us (2003us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572512.032633:0:109868:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 4 00000020:00010000:2.0:1788572512.045859:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_release():245 00000020:00010000:2.0:1788572512.045865:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.045873:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550fb0 (ffff92fc6721aea8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572512.045886:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.045902:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572512.045918:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572512.045921:0:109930:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550fb0 (ffff92fc6721ae18/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572512.045936:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572512.045948:0:109930:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988) current size 12288, valid attrs 100101f0, mode 92fc, uid 0, gid 0, new size 0, atime 1788572496.000000000, mtime 1788572501.000000000, ctime 1788572511.000000000, rc 0 00000080:00200080:2.0:1788572512.045979:0:109930:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=c00c66f3 arg=bae790 00010000:00010000:2.0:1788572512.046032:0:109930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.046049:0:109930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef3524fd lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0xc9a239a221792459 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.046069:0:109930:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617dc00 x1875449863853440/t4294983690(4294983690) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572527 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572512.046079:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572512.046085:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863856640/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 504/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572512.046107:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863856640:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572512.046424:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572512.051187:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.051505:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881866880 00000100:00100000:0.0:1788572512.051532:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881866880 00000100:00100000:0.0:1788572512.051601:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 5 00000100:00100000:0.0:1788572512.051612:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:125142:x1875449881866880:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572512.051630:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572512.051707:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572512.051714:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:125142:x1875449881866880:12345-192.168.202.132@tcp:104: Request processed in 104us (542us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572512.051761:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 5 00010000:00010000:2.0:1788572512.052099:0:109878:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572512.052108:0:109878:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.052117:0:109878:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204000 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572512.052122:0:109878:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.052130:0:109878:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.052137:0:109878:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 3/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572512.052174:0:109878:0:(vvp_object.c:117:vvp_conf_set()) [0x20000041d:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572512.052224:0:109878:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 2/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.052231:0:109878:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572512.052261:0:109878:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.052270:0:109878:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70204000/0x58a604dcef352519 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a2217924b4 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.052289:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.052290:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.052292:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572512.060632:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.060809:0:109867:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1788572512.063019:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881867008 00000100:00100000:0.0:1788572512.063034:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881867008 00000100:00100000:1.0:1788572512.063482:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 6 00000100:00100000:1.0:1788572512.063495:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc47f01450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:125142:x1875449881867008:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572512.063527:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35250b lrc: 2/0,0 mode: CR/CR res: [0x20000041f:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792498 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572512.063633:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572512.063640:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc47f01450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:22e19d3b-9675-4547-8eb5-08aa5a0df26e+5:125142:x1875449881867008:12345-192.168.202.132@tcp:104: Request processed in 148us (3020us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572512.063650:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 6 00000100:00100000:2.0:1788572512.082594:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863856640:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572512.082636:0:109930:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617dc00 x1875449863853440/t4294983690(4294983690) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572512.082678:0:109930:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000002:00010000:2.0:1788572512.082691:0:109930:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572512.082777:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.082788:0:109930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.082799:0:109930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.082831:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863857024:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.083095:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.087683:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863857024:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.087758:0:109930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.087776:0:109930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.087794:0:109930:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff92fc4617df80 x1875449863857024/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 376/584 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572512.087807:0:109930:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572512.087835:0:109930:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.088025:0:109930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 16 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572512.088030:0:109930:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572512.088049:0:109930:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef352527 00000080:00010000:2.0:1788572512.088065:0:109930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352527 from it ffff9e0383697bc8 00010000:00010000:2.0:1788572512.088070:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.088080:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.088106:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.088117:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572512.088125:0:109930:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000041d:0x2:0x0](00000000e0e21c90) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.088181:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.088189:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572512.088431:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572512.088446:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572512.088480:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572512.088495:0:109930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.088504:0:109930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572512.088585:0:109930:0:(file.c:6948:ll_layout_lock_set()) ### file [0x20000041d:0x2:0x0](00000000e0e21c90) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572512.088606:0:109930:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572512.088612:0:109930:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572512.088623:0:109930:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000008:00010000:2.0:1788572512.088635:0:109930:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 9000 blocks 24 [cma]time: 1788572501 1788572501 1788572496 by oinfo size 9000 blocks 24 [cma]time 1788572511 1788572501 1788572496 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572512.088785:0:109930:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 2 00010000:00010000:2.0:1788572512.088792:0:109930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.088799:0:109930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367800/0x58a604dcef352527 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217924d0 expref: -99 pid: 109930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788572512.089440:0:109930:0:(file.c:479:ll_file_release()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), flags 1154002 00000002:00100000:2.0:1788572512.089501:0:109930:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67288a80 x1875449863851776/t4294983686(4294983686) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 592/696 e 0 to 0 dl 1788572527 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572512.089588:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572512.089596:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863857280/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572512.089625:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863857280:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572512.089810:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.097699:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863857280:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572512.099315:0:109930:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67288a80 x1875449863851776/t4294983686(4294983686) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 592/696 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572512.099411:0:109930:0:(file.c:479:ll_file_release()) START file . :VOLATILE:0000:42BE3182:fd=03:[0x20000041f:0x1:0x0](ffff92fc665d9988), flags 554001 00000002:00100000:2.0:1788572512.099471:0:109930:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617c700 x1875449863852160/t4294983687(4294983687) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572527 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572512.099483:0:109930:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572512.099489:0:109930:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617df80 x1875449863857536/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572512.099512:0:109930:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863857536:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572512.099654:0:109930:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.106574:0:109930:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109930:1875449863857536:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572512.106611:0:109930:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617c700 x1875449863852160/t4294983687(4294983687) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572512.127518:0:109931:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.127551:0:109931:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.127573:0:109931:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572512.127589:0:109931:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572512.127599:0:109931:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.127607:0:109931:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.127610:0:109931:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.127619:0:109931:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836afbf8 00010000:00010000:2.0:1788572512.127623:0:109931:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572512.127632:0:109931:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572512.127647:0:109931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.127660:0:109931:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665dc2c8), flags=27 00000080:00200000:2.0:1788572512.127664:0:109931:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665dc2c8), intent=0 00000002:00010000:2.0:1788572512.127676:0:109931:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572512.127943:0:109931:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.127953:0:109931:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.127966:0:109931:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.127998:0:109931:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863857664:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.128615:0:109931:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.133670:0:109931:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863857664:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.133699:0:109931:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133764:0:109931:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133774:0:109931:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133787:0:109931:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133797:0:109931:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133819:0:109931:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133828:0:109931:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133836:0:109931:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.133856:0:109931:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617c700 x1875449863857664/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572512.133869:0:109931:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133879:0:109931:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:2.0:1788572512.133886:0:109931:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572512.133911:0:109931:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1788572512.133914:0:109931:0:(namei.c:147:ll_iget()) got inode: [0x20000041d:0x1:0x0](ffff92fc665d42c8): rc = 0 00000080:00010000:2.0:1788572512.133925:0:109931:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d42c8) for lock 0x58a604dcef35252e 00000080:00010000:2.0:1788572512.133936:0:109931:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d42c8) 00000080:00010000:2.0:1788572512.133940:0:109931:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35252e from it ffff9e03836afac8 00010000:00010000:2.0:1788572512.133943:0:109931:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.133951:0:109931:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70367e00/0x58a604dcef35252e lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217924f3 expref: -99 pid: 109931 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572512.133987:0:109931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.134123:0:109931:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), file ffff92fc464b9380, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572512.134132:0:109931:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=open 00000002:00010000:2.0:1788572512.134211:0:109931:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00000100:00100000:0.0:1788572512.140445:0:109931:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863857920:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572512.140471:0:109931:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140484:0:109931:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140491:0:109931:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140498:0:109931:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140507:0:109931:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140513:0:109931:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205a00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572512.140518:0:109931:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140525:0:109931:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140532:0:109931:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140539:0:109931:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572512.140546:0:109931:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.140553:0:109931:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205a00/0x58a604dcef352535 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572512.140578:0:109931:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc4617c700 x1875449863857920/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572512.140594:0:109931:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:0.0:1788572512.140805:0:109931:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:0.0:1788572512.140838:0:109931:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572512.141109:0:109931:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572512.141123:0:109931:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572512.141131:0:109931:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572512.141138:0:109931:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:0.0:1788572512.141142:0:109931:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:0.0:1788572512.141148:0:109931:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836afbf8 00010000:00010000:0.0:1788572512.141151:0:109931:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572512.141293:0:109931:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572512.141309:0:109931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788572512.141326:0:109931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:0.0:1788572512.141333:0:109931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 44 00000080:00200008:0.0:1788572512.141343:0:109931:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00000002:00010000:0.0:1788572512.141359:0:109931:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788572512.141405:0:109931:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.141412:0:109931:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572512.141424:0:109931:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572512.141433:0:109931:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67048380 x1875449863858176/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572512.141444:0:109931:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572512.141466:0:109931:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863858176:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572512.141706:0:109931:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572512.150129:0:109931:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863858176:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572512.150164:0:109931:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150177:0:109931:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150184:0:109931:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150192:0:109931:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150200:0:109931:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150218:0:109931:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572512.150222:0:109931:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150229:0:109931:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150235:0:109931:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150241:0:109931:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572512.150247:0:109931:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.150253:0:109931:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3200/0x58a604dcef35253c lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109931 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572512.150273:0:109931:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc67048380 x1875449863858176/t4294983698(4294983698) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572512.150286:0:109931:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048380 x1875449863858176/t4294983698(4294983698) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572512.150298:0:109931:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572512.150330:0:109931:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000002:00100000:0.0:1788572512.150352:0:109931:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048380 x1875449863858176/t4294983698(4294983698) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1788572512.150450:0:109931:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8) cmd=c0086915 arg=14fa620 00000100:00100000:0.0:1788572512.150548:0:109931:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863858304:192.168.202.132@tcp:34:lfs.0 00000100:00100000:0.0:1788572512.150631:0:109931:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572512.155077:0:109931:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863858304:192.168.202.132@tcp:34:lfs.0 00000080:00200008:0.0:1788572512.157437:0:109931:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00000002:00100000:0.0:1788572512.157529:0:109931:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67048380 x1875449863858176/t4294983698(4294983698) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572512.157544:0:109931:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572512.157550:0:109931:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704a680 x1875449863858560/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572512.157575:0:109931:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109931:1875449863858560:192.168.202.132@tcp:35:lfs.0 00000100:00100000:0.0:1788572512.157837:0:109931:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f480 going to sleep for 11 seconds 00000080:00200000:2.0:1788572512.174337:0:109932:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.174372:0:109932:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.174394:0:109932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572512.174411:0:109932:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572512.174420:0:109932:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.174428:0:109932:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.174432:0:109932:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.174439:0:109932:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e0383697bf8 00010000:00010000:2.0:1788572512.174444:0:109932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572512.174453:0:109932:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572512.174471:0:109932:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.174480:0:109932:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:2.0:1788572512.174485:0:109932:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.174497:0:109932:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), file ffff92fc471ea300, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572512.174504:0:109932:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=open 00000002:00010000:2.0:1788572512.174562:0:109932:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572512.174620:0:109932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.174630:0:109932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572512.174642:0:109932:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572512.174654:0:109932:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617dc00 x1875449863858816/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572512.174663:0:109932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.174688:0:109932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109932:1875449863858816:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.174922:0:109932:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.178531:0:109932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109932:1875449863858816:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.178558:0:109932:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178572:0:109932:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178580:0:109932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178587:0:109932:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178596:0:109932:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178602:0:109932:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366000 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572512.178607:0:109932:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178613:0:109932:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178621:0:109932:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178627:0:109932:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572512.178634:0:109932:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.178641:0:109932:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef352543 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.178660:0:109932:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc4617dc00 x1875449863858816/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572512.178679:0:109932:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572512.178870:0:109932:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.178892:0:109932:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.178907:0:109932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572512.178919:0:109932:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572512.178925:0:109932:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.178931:0:109932:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.178934:0:109932:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.178939:0:109932:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e0383697bf8 00010000:00010000:2.0:1788572512.178943:0:109932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572512.178952:0:109932:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572512.178965:0:109932:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.178971:0:109932:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:2.0:1788572512.178977:0:109932:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572512.178985:0:109932:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00000002:00010000:2.0:1788572512.178996:0:109932:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240304001 00010000:00010000:2.0:1788572512.179036:0:109932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.179048:0:109932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572512.179058:0:109932:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572512.179064:0:109932:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617dc00 x1875449863859072/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572512.179073:0:109932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 109932 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.179090:0:109932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109932:1875449863859072:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.179510:0:109932:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.183569:0:109932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109932:1875449863859072:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.183605:0:109932:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.183616:0:109932:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.183627:0:109932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.183637:0:109932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.183651:0:109932:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc4617dc00 x1875449863859072/t4294983700(4294983700) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572512.183668:0:109932:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617dc00 x1875449863859072/t4294983700(4294983700) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572512.183679:0:109932:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.183688:0:109932:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 16 res 8589935645/1 (0 0) 00000080:00200000:2.0:1788572512.183694:0:109932:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572512.183713:0:109932:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572512.183717:0:109932:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665d42c8) for lock 0x58a604dcef35254a 00000080:00010000:2.0:1788572512.183768:0:109932:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35254a from it ffff9e0383697bd0 00010000:00010000:2.0:1788572512.183772:0:109932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.183782:0:109932:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572512.183815:0:109932:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617dc00 x1875449863859072/t4294983700(4294983700) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572512.183929:0:109932:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8) cmd=c0086915 arg=d3a620 00000100:00100000:2.0:1788572512.184044:0:109932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109932:1875449863859200:192.168.202.132@tcp:34:lfs.0 00000100:00100000:2.0:1788572512.184124:0:109932:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.188529:0:109932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109932:1875449863859200:192.168.202.132@tcp:34:lfs.0 00000080:00200008:2.0:1788572512.188709:0:109932:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00010000:00010000:2.0:1788572512.189018:0:109932:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572512.201867:0:109933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.201904:0:109933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.201925:0:109933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572512.201942:0:109933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572512.201951:0:109933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.201959:0:109933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.201963:0:109933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.201971:0:109933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836b7bf8 00010000:00010000:2.0:1788572512.201976:0:109933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572512.201985:0:109933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572512.202002:0:109933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.202024:0:109933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:2.0:1788572512.202031:0:109933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.202043:0:109933:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), file ffff92fc608bf680, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572512.202050:0:109933:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=open 00000002:00010000:2.0:1788572512.202175:0:109933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572512.202312:0:109933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.202336:0:109933:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572512.202348:0:109933:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572512.202363:0:109933:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863859456/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572512.202373:0:109933:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.202397:0:109933:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109933:1875449863859456:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.202595:0:109933:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.206547:0:109933:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109933:1875449863859456:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.206569:0:109933:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206582:0:109933:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206589:0:109933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206594:0:109933:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206603:0:109933:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206608:0:109933:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572512.206613:0:109933:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206619:0:109933:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206626:0:109933:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206632:0:109933:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572512.206638:0:109933:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.206645:0:109933:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352551 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 109933 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.206662:0:109933:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc4617d180 x1875449863859456/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572512.206676:0:109933:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572512.206956:0:109933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.206977:0:109933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.206991:0:109933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572512.207002:0:109933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572512.207122:0:109933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.207128:0:109933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.207132:0:109933:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.207138:0:109933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836b7bf8 00010000:00010000:2.0:1788572512.207141:0:109933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572512.207149:0:109933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572512.207164:0:109933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.207172:0:109933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:2.0:1788572512.207179:0:109933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572512.207189:0:109933:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00000080:00200080:2.0:1788572512.207297:0:109933:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8) cmd=c0086915 arg=7f2620 00000100:00100000:2.0:1788572512.207392:0:109933:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109933:1875449863859584:192.168.202.132@tcp:34:lfs.0 00000100:00100000:2.0:1788572512.207511:0:109933:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.211381:0:109933:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109933:1875449863859584:192.168.202.132@tcp:34:lfs.0 00000080:00200008:2.0:1788572512.211717:0:109933:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00010000:00010000:2.0:1788572512.211786:0:109933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572512.225570:0:109934:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.225605:0:109934:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.225626:0:109934:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572512.225642:0:109934:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572512.225657:0:109934:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.225665:0:109934:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.225669:0:109934:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.225678:0:109934:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836bfbf8 00010000:00010000:2.0:1788572512.225681:0:109934:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572512.225690:0:109934:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572512.225708:0:109934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.225716:0:109934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:2.0:1788572512.225800:0:109934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.225882:0:109934:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), file ffff92fc49280180, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572512.225890:0:109934:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=open 00000002:00010000:2.0:1788572512.225939:0:109934:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572512.225991:0:109934:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.226000:0:109934:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572512.226012:0:109934:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572512.226022:0:109934:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863859840/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572512.226031:0:109934:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.226053:0:109934:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109934:1875449863859840:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.226428:0:109934:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.229676:0:109934:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109934:1875449863859840:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.229700:0:109934:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229714:0:109934:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229752:0:109934:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229761:0:109934:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229770:0:109934:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229776:0:109934:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572512.229781:0:109934:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229788:0:109934:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229796:0:109934:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229804:0:109934:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572512.229824:0:109934:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.229830:0:109934:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352558 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 109934 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.229853:0:109934:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc4617d180 x1875449863859840/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572512.229866:0:109934:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572512.230010:0:109934:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.230030:0:109934:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.230045:0:109934:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572512.230067:0:109934:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572512.230075:0:109934:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.230081:0:109934:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.230085:0:109934:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.230091:0:109934:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836bfbf8 00010000:00010000:2.0:1788572512.230095:0:109934:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572512.230105:0:109934:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572512.230121:0:109934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.230129:0:109934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:2.0:1788572512.230136:0:109934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572512.230145:0:109934:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00000080:00200080:2.0:1788572512.230184:0:109934:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8) cmd=c0086915 arg=1221620 00000100:00100000:2.0:1788572512.230468:0:109934:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109934:1875449863859968:192.168.202.132@tcp:34:lfs.0 00000100:00100000:2.0:1788572512.230562:0:109934:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.243802:0:109934:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109934:1875449863859968:192.168.202.132@tcp:34:lfs.0 00000080:00200008:2.0:1788572512.244017:0:109934:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags 304000 00010000:00010000:2.0:1788572512.244051:0:109934:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366a00/0x58a604dcef35254a lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179254e expref: -99 pid: 109932 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572512.266392:0:109935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:0.0:1788572512.266425:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572512.266446:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788572512.266465:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788572512.266473:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572512.266482:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:0.0:1788572512.266486:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:0.0:1788572512.266495:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836c7b58 00010000:00010000:0.0:1788572512.266499:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788572512.266508:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788572512.266526:0:109935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788572512.266533:0:109935:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:0.0:1788572512.266539:0:109935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788572512.266552:0:109935:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), flags=9 00000080:00200000:0.0:1788572512.266557:0:109935:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665d42c8), intent=getattr 00000002:00010000:0.0:1788572512.266604:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572512.266655:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.266664:0:109935:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.266675:0:109935:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572512.266702:0:109935:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863860096:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572512.268552:0:109935:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.271228:0:109935:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863860096:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.271257:0:109935:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271275:0:109935:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271283:0:109935:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271294:0:109935:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271306:0:109935:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271329:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271339:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271346:0:109935:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.271360:0:109935:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704aa00 x1875449863860096/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 592/856 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572512.271371:0:109935:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572512.271384:0:109935:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271394:0:109935:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 4 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572512.271401:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572512.271461:0:109935:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00200000:2.0:1788572512.271465:0:109935:0:(namei.c:147:ll_iget()) got inode: [0x20000041d:0x2:0x0](ffff92fc665d1988): rc = 0 00000080:00200000:2.0:1788572512.271481:0:109935:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 2 -> 2 00000080:00010000:2.0:1788572512.271491:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:2.0:1788572512.271503:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572512.271507:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e03836c7b20 00010000:00010000:2.0:1788572512.271510:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271519:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572512.271565:0:109935:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572512.271572:0:109935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572512.271627:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.271637:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.271647:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572512.271654:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.271659:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572512.271663:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:2.0:1788572512.271667:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e03836c7cb8 00010000:00010000:2.0:1788572512.271670:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.271679:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572512.271771:0:109935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:2.0:1788572512.271780:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.271789:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572512.271796:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572512.271807:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.271821:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:2.0:1788572512.271823:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:2.0:1788572512.271827:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e03836c7bf8 00010000:00010000:2.0:1788572512.271829:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572512.271837:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572512.271845:0:109935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.271851:0:109935:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:2.0:1788572512.271856:0:109935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572512.271859:0:109935:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, flags=265 00000080:00200000:2.0:1788572512.271873:0:109935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), inode mode 81a4 mask 44 00000080:00200008:2.0:1788572512.271882:0:109935:0:(file.c:1090:ll_file_open()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), flags 154400 00000002:00010000:2.0:1788572512.271901:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0100240114401 00010000:00010000:2.0:1788572512.271948:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.271955:0:109935:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572512.271966:0:109935:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572512.271975:0:109935:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863860352/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572512.271983:0:109935:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.272005:0:109935:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863860352:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.272183:0:109935:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.277680:0:109935:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863860352:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.277707:0:109935:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277754:0:109935:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277763:0:109935:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277775:0:109935:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277786:0:109935:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277799:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277807:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277825:0:109935:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.277843:0:109935:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617d180 x1875449863860352/t4294983701(4294983701) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572512.277855:0:109935:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863860352/t4294983701(4294983701) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572512.277867:0:109935:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572512.277882:0:109935:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277895:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.277906:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.277914:0:109935:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277920:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277926:0:109935:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.277951:0:109935:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572512.277959:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572512.278004:0:109935:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00200000:2.0:1788572512.278019:0:109935:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 2 -> 2 00000080:00010000:2.0:1788572512.278024:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:2.0:1788572512.278031:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e03836c7bf0 00010000:00010000:2.0:1788572512.278034:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572512.278043:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1788572512.278079:0:109935:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863860352/t4294983701(4294983701) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572512.278185:0:109935:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=c00c66f3 arg=7ffea0b0a024 00010000:00010000:1.0:1788572512.278396:0:109878:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572512.278404:0:109878:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278411:0:109878:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:1.0:1788572512.278416:0:109878:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278425:0:109878:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278432:0:109878:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 3/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278451:0:109878:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572512.278488:0:109878:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 2/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278495:0:109878:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572512.278521:0:109878:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278529:0:109878:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366400/0x58a604dcef352566 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217925e8 expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.278549:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572512.278551:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572512.278552:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572512.278560:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572512.278576:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863860480:192.168.202.132@tcp:103:ldlm_bl.0 00000002:00010000:2.0:1788572512.279143:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: open flags 01004040000001 00010000:00010000:2.0:1788572512.279185:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.279192:0:109935:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572512.279202:0:109935:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572512.279219:0:109935:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863860736/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572512.279228:0:109935:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x81000000000000 pid: 109935 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572512.279243:0:109935:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863860736:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572512.279389:0:109935:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fb40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572512.283576:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863860480:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572512.283844:0:109935:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863860736:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572512.283867:0:109935:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283879:0:109935:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 4/1,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283891:0:109935:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283903:0:109935:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283916:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283924:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283931:0:109935:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 4/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.283941:0:109935:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff92fc4617c380 x1875449863860736/t4294983702(4294983702) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572512.283954:0:109935:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c380 x1875449863860736/t4294983702(4294983702) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572512.283970:0:109935:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.283983:0:109935:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5dd91800 type 13 mode 2 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572512.283990:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1788572512.284139:0:109935:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c380 x1875449863860736/t4294983702(4294983702) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572512.284149:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35256d 00000080:00010000:2.0:1788572512.284156:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35256d from it ffff9e03836c7cc0 00010000:00010000:2.0:1788572512.284160:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.284168:0:109935:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572512.284477:0:109935:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), inode mode 81a4 mask 4 00000080:00200000:2.0:1788572512.284487:0:109935:0:(xattr.c:744:ll_xattr_get()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), xattr lov 00000080:00200008:2.0:1788572512.284519:0:109935:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572512.284527:0:109935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572512.284577:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.284588:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572512.284599:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572512.284610:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.284616:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572512.284620:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:2.0:1788572512.284623:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e03836c7cd8 00010000:00010000:2.0:1788572512.284626:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572512.284634:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200080:2.0:1788572512.285283:0:109935:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=66f4 arg=1 00000080:00200080:2.0:1788572512.320701:0:109935:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=1 00000080:00200008:2.0:1788572512.320744:0:109935:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572512.320754:0:109935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572512.320884:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572512.320907:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572512.320924:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572512.320932:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572512.320941:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:2.0:1788572512.320944:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:2.0:1788572512.320954:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e03836c7cd8 00010000:00010000:2.0:1788572512.320958:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572512.320968:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00000080:00200008:2.0:1788572512.321000:0:109935:0:(file.c:2483:do_file_read_iter()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], ppos: 0, count: 9000 00000080:00200000:2.0:1788572512.321209:0:109935:0:(file.c:2090:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: read ppos: 0, bytes: 9000 00000080:00200000:2.0:1788572512.321220:0:109935:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 1 00000020:00200000:2.0:1788572512.321229:0:109935:0:(cl_io.c:214:cl_io_rw_init()) header@00000000cb670240[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572512.321231:0:109935:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 9000) 2048 0 00000080:00200000:2.0:1788572512.321237:0:109935:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 1 restore needed 0 00000080:00200000:2.0:1788572512.321253:0:109935:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:2.0:1788572512.321262:0:109935:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 9000) 9000, 9000 00020000:00200000:2.0:1788572512.321267:0:109935:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572512.321284:0:109935:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 2] 00000008:00010000:2.0:1788572512.321298:0:109935:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:2.0:1788572512.321302:0:109935:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550c30, osc lock ffff92fc58370be0, flags 0x40000 00000080:00200000:2.0:1788572512.321310:0:109935:0:(vvp_io.c:835:vvp_io_read_start()) crhIBIW0-TeUOeG6ka7IzA: read [0, 9000) 00000001:00010000:2.0:1788572512.321333:0:109935:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572512.321340:0:109935:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572512.321343:0:109935:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550a30, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572512.321358:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.321370:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 2/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572512.321385:0:109935:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572512.321401:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572512.321405:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572512.321412:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788572512.321423:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572512.321430:0:109935:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572512.321434:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70474b90 (ffff92fc6721ae18/0) at cl_lock_release():245 00000020:00010000:2.0:1788572512.321437:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70474b90 (ffff92fc6721ae18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.321440:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550a30 (ffff92fc6721aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.321444:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70474b90 (ffff92fc6721ae18/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572512.321447:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550a30 (ffff92fc6721ae18/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572512.321458:0:109935:0:(vvp_io.c:893:vvp_io_read_start()) count:9000 iocb pos:0 00000080:00200000:2.0:1788572512.321464:0:109935:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), size=9000 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel, unaligned 00000020:00200000:2.0:1788572512.321522:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:2.0:1788572512.321525:0:109935:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 9000 count 9000 file_offset 9000 00000020:00010000:2.0:1788572512.321533:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_release():245 00000020:00010000:2.0:1788572512.321536:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.321539:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550c30 (ffff92fc6721aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.321542:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572512.321544:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550c30 (ffff92fc6721ae18/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572512.321550:0:109935:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000020:00200000:2.0:1788572512.321556:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572512.321583:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc451f5900 00000008:00100000:2.0:1788572512.321635:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617f800 - object 0x2c0000401:537 offset 0<>9000 00000100:00100000:2.0:1788572512.321716:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863860864:192.168.202.132@tcp:3:lfs.0 00000100:00100000:1.0:1788572512.325381:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.325495:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881867648 00000100:00100000:0.0:1788572512.325507:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881867648 00000100:00100000:0.0:1788572512.325598:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 7 00000100:00100000:0.0:1788572512.325610:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6728aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8450:x1875449881867648:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572512.325628:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 00000000ef523264/0x58a604dcef352520 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000020000 nid: local remote: 0xc9a239a2217924bb expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572512.325764:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572512.325772:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6728aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:8450:x1875449881867648:12345-192.168.202.132@tcp:104: Request processed in 165us (403us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572512.325780:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 7 00000100:00100000:2.0:1788572512.326334:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863860992:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572512.331896:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863860992:192.168.202.132@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1788572512.331967:0:109935:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:1.0:1788572512.331981:0:109935:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572512.331986:0:109935:0:(file.c:2238:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: 1 io complete with rc: 0, result: 9000, restart: 0 00000020:00200000:1.0:1788572512.331989:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572512.331997:0:109935:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200080:1.0:1788572512.332125:0:109935:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=0 00000080:00200080:1.0:1788572512.332151:0:109935:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=2 00000080:00200008:1.0:1788572512.332159:0:109935:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572512.332167:0:109935:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:1.0:1788572512.332225:0:109935:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572512.332244:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572512.332260:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788572512.332268:0:109935:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572512.332274:0:109935:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:1.0:1788572512.332278:0:109935:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:1.0:1788572512.332286:0:109935:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e03836c7cd8 00010000:00010000:1.0:1788572512.332291:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572512.332301:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00000080:00200008:1.0:1788572512.332339:0:109935:0:(file.c:2483:do_file_read_iter()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], ppos: 0, count: 9000 00000080:00200000:1.0:1788572512.332348:0:109935:0:(file.c:2090:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: read ppos: 0, bytes: 9000 00000080:00200000:1.0:1788572512.332356:0:109935:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 2 00000020:00200000:1.0:1788572512.332363:0:109935:0:(cl_io.c:214:cl_io_rw_init()) header@00000000cb670240[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:1.0:1788572512.332365:0:109935:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 9000) 2048 0 00000080:00200000:1.0:1788572512.332370:0:109935:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:1.0:1788572512.332384:0:109935:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:1.0:1788572512.332391:0:109935:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 9000) 9000, 9000 00020000:00200000:1.0:1788572512.332395:0:109935:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788572512.332409:0:109935:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 2] 00000008:00010000:1.0:1788572512.332423:0:109935:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:1.0:1788572512.332427:0:109935:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4e30, osc lock ffff92fc47ff44c0, flags 0x40000 00000080:00200000:1.0:1788572512.332434:0:109935:0:(vvp_io.c:835:vvp_io_read_start()) crhIBIW0-TeUOeG6ka7IzA: read [0, 9000) 00000001:00010000:1.0:1788572512.332444:0:109935:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:1.0:1788572512.332449:0:109935:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572512.332451:0:109935:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4cb0, osc lock ffff92fc47ff4980, flags 0x400001002 00010000:00010000:1.0:1788572512.332465:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572512.332476:0:109935:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572512.332485:0:109935:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572512.332498:0:109935:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572512.332536:0:109935:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.332546:0:109935:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572512.332583:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788572512.332660:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572512.332677:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863861120:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572512.335502:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.335606:0:109868:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881867776 00000100:00100000:0.0:1788572512.335616:0:109868:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881867776 00000100:00100000:0.0:1788572512.335712:0:109868:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 8 00000100:00100000:0.0:1788572512.335744:0:109868:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:130898:x1875449881867776:12345-192.168.202.132@tcp:106: 00010000:00010000:0.0:1788572512.335760:0:109868:0:(ldlm_lockd.c:2525:ldlm_callback_handler()) ### glimpse ast ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572512.335774:0:109868:0:(ldlm_lockd.c:2116:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:0.0:1788572512.335806:0:109868:0:(cl_object.c:322:cl_object_glimpse()) header@00000000ddb15274[0x110, 1, [0x20000041f:0x1:0x0] hash] 00000020:00010000:0.0:1788572512.335808:0:109868:0:(cl_object.c:322:cl_object_glimpse()) size: 9000 mtime: 1788572501 atime: 1788572496 ctime: 1788572511 blocks: 24 00000020:00010000:0.0:1788572512.335824:0:109868:0:(cl_object.c:322:cl_object_glimpse()) header@00000000a6015f2b[0x116, 2, [0x280000401:0x228:0x0]] 00000020:00010000:0.0:1788572512.335826:0:109868:0:(cl_object.c:322:cl_object_glimpse()) size: 9000 mtime: 1788572501 atime: 1788572496 ctime: 1788572511 blocks: 24 00000100:00100000:0.0:1788572512.335895:0:109868:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:130898:x1875449881867776:12345-192.168.202.132@tcp:106: Request processed in 174us (402us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572512.335904:0:109868:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 8 00010000:00010000:0.0:1788572512.338546:0:2393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338569:0:2393:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338584:0:2393:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338592:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338600:0:2393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 5/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572512.338614:0:2393:0:(osc_lock.c:187:osc_lock_lvb_update()) ### no dlm lock: acquired size 9000, setting rss=9000; leaving kms=0, end=18446744073709551615 00000020:00200000:0.0:1788572512.338620:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788572512.338679:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 4/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338689:0:2393:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338698:0:2393:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 4/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338705:0:2393:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f4600 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:0.0:1788572512.338711:0:2393:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 5/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338759:0:2393:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572512.338767:0:2393:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 3/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.338775:0:2393:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000073987950/0x58a604dcef352574 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 109935 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572512.338798:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863861120:192.168.202.132@tcp:101:lfs.0 00000080:00200000:1.0:1788572512.338958:0:109935:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 48 00000020:00010000:1.0:1788572512.338965:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70474b90 (ffff92fc6721ae18/0) at cl_lock_release():245 00000020:00010000:1.0:1788572512.338968:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70474b90 (ffff92fc6721ae18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572512.338974:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4cb0 (ffff92fc6721aea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572512.338977:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70474b90 (ffff92fc6721ae18/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572512.338980:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4cb0 (ffff92fc6721ae18/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572512.338993:0:109935:0:(vvp_io.c:893:vvp_io_read_start()) count:9000 iocb pos:0 00000080:00200000:1.0:1788572512.338999:0:109935:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), size=9000 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel, unaligned 00000020:00200000:1.0:1788572512.339181:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:1.0:1788572512.339184:0:109935:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 9000 count 9000 file_offset 9000 00000020:00010000:1.0:1788572512.339192:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_release():245 00000020:00010000:1.0:1788572512.339194:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572512.339197:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4e30 (ffff92fc6721aea8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572512.339200:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3a88 (ffff92fc6721ae18/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572512.339203:0:109935:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4e30 (ffff92fc6721ae18/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572512.339209:0:109935:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000020:00200000:1.0:1788572512.339217:0:109935:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1788572512.339235:0:103957:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc482276c0 00000008:00100000:1.0:1788572512.339281:0:103957:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc6728bb80 - object 0x280000401:552 offset 0<>9000 00000100:00100000:0.0:1788572512.339629:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863861248:192.168.202.132@tcp:3:lfs.0 00000100:00100000:1.0:1788572512.342645:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572512.342806:0:109867:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881867904 00000100:00100000:0.0:1788572512.342943:0:109867:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881867904 00000100:00100000:0.0:1788572512.343035:0:109867:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 9 00000100:00100000:0.0:1788572512.343046:0:109867:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67289880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:123914:x1875449881867904:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572512.343128:0:109867:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572512.343256:0:109867:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572512.343264:0:109867:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67289880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:123914:x1875449881867904:12345-192.168.202.132@tcp:104: Request processed in 220us (628us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572512.343272:0:109867:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 9 00010000:00010000:2.0:1788572512.343535:0:109878:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572512.343543:0:109878:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572512.343552:0:109878:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366e00 already unused, calling callback (ffffffffc10a1300) 00010000:00010000:2.0:1788572512.343558:0:109878:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572512.343581:0:109878:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041f:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572512.343594:0:109878:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788572512.343634:0:109878:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 2/0,0 mode: --/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572512.343641:0:109878:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572512.343824:0:109878:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 1/0,0 mode: --/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572512.343834:0:109878:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 00000000aeb69306/0x58a604dcef352512 lrc: 0/0,0 mode: --/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0xc9a239a22179249f expref: -99 pid: 109930 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572512.343872:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.343874:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.343876:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1788572512.351550:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc482276c0 00000020:00200000:0.0:1788572512.351563:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:0.0:1788572512.351566:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:0.0:1788572512.351568:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1788572512.351574:0:2393:0:(cl_io.c:1302:cl_sub_dio_end()) finishing unaligned dio read aio->cda_bytes 9000 00000020:00200000:0.0:1788572512.351583:0:2393:0:(cl_io.c:1693:__ll_dio_user_copy()) count 9000, offset 0, pos 0, cdp_page_count 3 00000020:00200000:0.0:1788572512.351589:0:2393:0:(cl_io.c:1693:__ll_dio_user_copy()) count 4904, offset 0, pos 4096, cdp_page_count 3 00000020:00200000:0.0:1788572512.351592:0:2393:0:(cl_io.c:1693:__ll_dio_user_copy()) count 808, offset 0, pos 8192, cdp_page_count 3 00000020:00200000:0.0:1788572512.351601:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1788572512.351624:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc482276c0 00000100:00100000:0.0:1788572512.351650:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863861248:192.168.202.132@tcp:3:lfs.0 00000100:00100000:2.0:1788572512.356360:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863861376:192.168.202.132@tcp:103:ldlm_bl.0 00000080:00200080:2.0:1788572512.370212:0:109935:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=c00c66f3 arg=7ffea0b0a034 00010000:00010000:2.0:1788572512.370245:0:109935:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.370302:0:109935:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 1/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572512.370311:0:109935:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572512.370345:0:109935:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863861504:192.168.202.132@tcp:103:lfs.0 00000100:00100000:2.0:1788572512.370449:0:109935:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.373936:0:109935:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863861504:192.168.202.132@tcp:103:lfs.0 00010000:00010000:2.0:1788572512.373968:0:109935:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70366000/0x58a604dcef35256d lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a2217925fd expref: -99 pid: 109935 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.374149:0:109935:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617c380 x1875449863860736/t4294983702(4294983702) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572512.374162:0:109935:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572512.374169:0:109935:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617f800 x1875449863861760/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572512.374192:0:109935:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863861760:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572512.374336:0:109935:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.377267:0:109935:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863861760:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572512.377304:0:109935:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617c380 x1875449863860736/t4294983702(4294983702) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572512.377367:0:109935:0:(file.c:479:ll_file_release()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), flags 154000 00000002:00100000:2.0:1788572512.377420:0:109935:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617d180 x1875449863860352/t4294983701(4294983701) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572512.377430:0:109935:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572512.377435:0:109935:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617f800 x1875449863862016/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572512.377454:0:109935:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863862016:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572512.377573:0:109935:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.384266:0:109935:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109935:1875449863862016:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572512.384304:0:109935:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617d180 x1875449863860352/t4294983701(4294983701) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572512.423051:0:109937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:1.0:1788572512.423083:0:109937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572512.423105:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572512.423123:0:109937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572512.423132:0:109937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572512.423142:0:109937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:1.0:1788572512.423146:0:109937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:1.0:1788572512.423154:0:109937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e0383697b58 00010000:00010000:1.0:1788572512.423159:0:109937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572512.423167:0:109937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788572512.423183:0:109937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572512.423191:0:109937:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:1.0:1788572512.423197:0:109937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572512.423202:0:109937:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, flags=9 00000080:00200008:1.0:1788572512.423240:0:109937:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572512.423246:0:109937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:1.0:1788572512.423270:0:109937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572512.423278:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572512.423287:0:109937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572512.423292:0:109937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572512.423296:0:109937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:1.0:1788572512.423298:0:109937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:1.0:1788572512.423302:0:109937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e0383697cb8 00010000:00010000:1.0:1788572512.423305:0:109937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572512.423311:0:109937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572512.423352:0:109937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:1.0:1788572512.423359:0:109937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572512.423366:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788572512.423374:0:109937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788572512.423381:0:109937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572512.423385:0:109937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:1.0:1788572512.423388:0:109937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef3524c5 00000080:00010000:1.0:1788572512.423392:0:109937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3524c5 from it ffff9e0383697bf8 00010000:00010000:1.0:1788572512.423395:0:109937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788572512.423401:0:109937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f4400/0x58a604dcef3524c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217923c6 expref: -99 pid: 109930 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788572512.423409:0:109937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572512.423413:0:109937:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:1.0:1788572512.423417:0:109937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665d42c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572512.423420:0:109937:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, flags=265 00000080:00200000:1.0:1788572512.423433:0:109937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), inode mode 81a4 mask 44 00000080:00200008:1.0:1788572512.423440:0:109937:0:(file.c:1090:ll_file_open()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), flags 154400 00000002:00010000:1.0:1788572512.423461:0:109937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0104240114401 00010000:00010000:1.0:1788572512.423511:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5000/0x58a604dcef35257b lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 109937 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.423519:0:109937:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5000/0x58a604dcef35257b lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109937 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572512.423532:0:109937:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572512.423546:0:109937:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67288000 x1875449863862272/t0(0) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572512.423555:0:109937:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5000/0x58a604dcef35257b lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 109937 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572512.423576:0:109937:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109937:1875449863862272:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572512.423781:0:109937:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329180 going to sleep for 11 seconds 00000080:00200080:0.0:1788572512.521890:0:109937:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=1 00000080:00200008:0.0:1788572512.521915:0:109937:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572512.521925:0:109937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:0.0:1788572512.522008:0:109937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572512.522029:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572512.522046:0:109937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 1 00000080:00200000:0.0:1788572512.522054:0:109937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572512.522062:0:109937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:0.0:1788572512.522066:0:109937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:0.0:1788572512.522073:0:109937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e0383697cd8 00010000:00010000:0.0:1788572512.522077:0:109937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572512.522086:0:109937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00000080:00200008:0.0:1788572512.522120:0:109937:0:(file.c:2483:do_file_read_iter()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], ppos: 0, count: 12288 00000080:00200000:0.0:1788572512.522133:0:109937:0:(file.c:2090:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: read ppos: 0, bytes: 12288 00000080:00200000:0.0:1788572512.522143:0:109937:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 1 00000020:00200000:0.0:1788572512.522151:0:109937:0:(cl_io.c:214:cl_io_rw_init()) header@00000000cb670240[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:0.0:1788572512.522153:0:109937:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 12288) 2048 0 00000080:00200000:0.0:1788572512.522157:0:109937:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788572512.522172:0:109937:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:0.0:1788572512.522179:0:109937:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 12288) 12288, 12288 00020000:00200000:0.0:1788572512.522183:0:109937:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572512.522198:0:109937:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 2] 00000008:00010000:0.0:1788572512.522212:0:109937:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:0.0:1788572512.522216:0:109937:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2d30, osc lock ffff92fc45932be0, flags 0x40000 00000080:00200000:0.0:1788572512.522222:0:109937:0:(vvp_io.c:835:vvp_io_read_start()) crhIBIW0-TeUOeG6ka7IzA: read [0, 12288) 00000001:00010000:0.0:1788572512.522231:0:109937:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:0.0:1788572512.522235:0:109937:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572512.522237:0:109937:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f22b0, osc lock ffff92fc45933690, flags 0x400001002 00010000:00010000:0.0:1788572512.522285:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.522294:0:109937:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572512.524785:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863862656:192.168.202.132@tcp:101:lfs.0 00000020:00200000:0.0:1788572512.525234:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572512.529385:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.529415:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.529430:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.530086:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572512.530258:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572512.530270:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788572512.530342:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.530352:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.530359:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.530369:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572512.530390:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863862656:192.168.202.132@tcp:101:lfs.0 00000080:00200000:2.0:1788572512.534610:0:109937:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 48 00000020:00010000:2.0:1788572512.534626:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572512.534631:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.534637:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f22b0 (ffff92fc6721a2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.534643:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572512.534646:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f22b0 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572512.534665:0:109937:0:(vvp_io.c:893:vvp_io_read_start()) count:12288 iocb pos:0 00000080:00200000:2.0:1788572512.534673:0:109937:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), size=12288 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel 00000020:00200000:2.0:1788572512.534757:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:2.0:1788572512.534760:0:109937:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 12288 count 12288 file_offset 12288 00000020:00010000:2.0:1788572512.534767:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3308 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572512.534769:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3308 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.534772:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2d30 (ffff92fc6721a2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572512.534774:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3308 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572512.534777:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2d30 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572512.534783:0:109937:0:(vvp_io.c:533:vvp_io_advance()) advancing 12288 bytes 00000020:00200000:2.0:1788572512.534789:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572512.534825:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc451f56c0 00000008:00100000:2.0:1788572512.534881:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617f800 - object 0x2c0000401:537 offset 0<>12288 00000100:00100000:2.0:1788572512.534928:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572512.534949:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863862784:192.168.202.132@tcp:3:lfs.0 00000008:00010000:2.0:1788572512.542076:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc451f56c0 00000020:00200000:2.0:1788572512.542091:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:2.0:1788572512.542106:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:2.0:1788572512.542108:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1788572512.542118:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572512.542143:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc451f56c0 00000100:00100000:2.0:1788572512.542175:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2395:1875449863862784:192.168.202.132@tcp:3:lfs.0 00000080:00200000:1.0:1788572512.542715:0:109937:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:1.0:1788572512.542766:0:109937:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572512.542772:0:109937:0:(file.c:2238:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: 1 io complete with rc: 0, result: 12288, restart: 0 00000020:00200000:1.0:1788572512.542775:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572512.542785:0:109937:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 12288 00000080:00200080:1.0:1788572512.542803:0:109937:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=0 00000080:00200080:1.0:1788572512.542870:0:109937:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=2 00000080:00200008:1.0:1788572512.542880:0:109937:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572512.542888:0:109937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:1.0:1788572512.542953:0:109937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572512.542976:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572512.542994:0:109937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572512.543003:0:109937:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572512.543020:0:109937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) 00000080:00010000:1.0:1788572512.543024:0:109937:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665d1988) for lock 0x58a604dcef35255f 00000080:00010000:1.0:1788572512.543033:0:109937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35255f from it ffff9e0383697cd8 00010000:00010000:1.0:1788572512.543036:0:109937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572512.543047:0:109937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc3a00/0x58a604dcef35255f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217925da expref: -99 pid: 109935 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000080:00200008:1.0:1788572512.543075:0:109937:0:(file.c:2483:do_file_read_iter()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0], ppos: 0, count: 12288 00000080:00200000:1.0:1788572512.543083:0:109937:0:(file.c:2090:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: read ppos: 0, bytes: 12288 00000080:00200000:1.0:1788572512.543222:0:109937:0:(file.c:1848:ll_io_set_mirror()) crhIBIW0-TeUOeG6ka7IzA: desiginated mirror: 2 00000020:00200000:1.0:1788572512.543230:0:109937:0:(cl_io.c:214:cl_io_rw_init()) header@00000000cb670240[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:1.0:1788572512.543232:0:109937:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 12288) 2048 0 00000080:00200000:1.0:1788572512.543236:0:109937:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:1.0:1788572512.543250:0:109937:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:1.0:1788572512.543257:0:109937:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 12288) 12288, 12288 00020000:00200000:1.0:1788572512.543261:0:109937:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788572512.543276:0:109937:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 2] 00000008:00010000:1.0:1788572512.543292:0:109937:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:1.0:1788572512.543297:0:109937:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4e30, osc lock ffff92fc47ff44c0, flags 0x40000 00000080:00200000:1.0:1788572512.543305:0:109937:0:(vvp_io.c:835:vvp_io_read_start()) crhIBIW0-TeUOeG6ka7IzA: read [0, 12288) 00000001:00010000:1.0:1788572512.543314:0:109937:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:1.0:1788572512.543329:0:109937:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572512.543331:0:109937:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4cb0, osc lock ffff92fc47ff4980, flags 0x400001002 00010000:00010000:1.0:1788572512.543388:0:109937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.543399:0:109937:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572512.543434:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788572512.543708:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788572512.543743:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863862912:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572512.548232:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.548257:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.548272:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.548290:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572512.548305:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572512.548326:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788572512.548342:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.548351:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.548359:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572512.548368:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572512.548387:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2394:1875449863862912:192.168.202.132@tcp:101:lfs.0 00000080:00200000:0.0:1788572512.548599:0:109937:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 48 00000020:00010000:0.0:1788572512.548606:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572512.548609:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572512.548615:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4cb0 (ffff92fc6721a2d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572512.548619:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704779d0 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572512.548622:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4cb0 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572512.548635:0:109937:0:(vvp_io.c:893:vvp_io_read_start()) count:12288 iocb pos:0 00000080:00200000:0.0:1788572512.548641:0:109937:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988), size=12288 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel 00000020:00200000:0.0:1788572512.548694:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:0.0:1788572512.548697:0:109937:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 12288 count 12288 file_offset 12288 00000020:00010000:0.0:1788572512.548704:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3308 (ffff92fc6721a7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1788572512.548707:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3308 (ffff92fc6721a7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572512.548709:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4e30 (ffff92fc6721a2d8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572512.548712:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3308 (ffff92fc6721a7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572512.548714:0:109937:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4e30 (ffff92fc6721a7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572512.548750:0:109937:0:(vvp_io.c:533:vvp_io_advance()) advancing 12288 bytes 00000020:00200000:0.0:1788572512.548758:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1788572512.548777:0:108280:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc4133b840 00000008:00100000:0.0:1788572512.548829:0:108280:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc67048a80 - object 0x280000401:552 offset 0<>12288 00000100:00100000:1.0:1788572512.549588:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572512.549609:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863863040:192.168.202.132@tcp:3:lfs.0 00000008:00010000:1.0:1788572512.568667:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc4133b840 00000020:00200000:1.0:1788572512.568680:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:1.0:1788572512.568683:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:1.0:1788572512.568684:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1788572512.568692:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788572512.568809:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc4133b840 00000100:00100000:1.0:1788572512.568840:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2393:1875449863863040:192.168.202.132@tcp:3:lfs.0 00000080:00200000:0.0:1788572512.575585:0:109937:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:0.0:1788572512.575609:0:109937:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572512.575616:0:109937:0:(file.c:2238:ll_file_io_generic()) crhIBIW0-TeUOeG6ka7IzA: 1 io complete with rc: 0, result: 12288, restart: 0 00000020:00200000:0.0:1788572512.575621:0:109937:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572512.575638:0:109937:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 12288 00000080:00200080:0.0:1788572512.575657:0:109937:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=400866b1 arg=0 00000080:00200080:0.0:1788572512.606262:0:109937:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d1988) cmd=c00c66f3 arg=7ffc65790114 00010000:00010000:0.0:1788572512.606290:0:109937:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205e00/0x58a604dcef352582 lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a22179263c expref: -99 pid: 109937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.609033:0:109937:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205e00/0x58a604dcef352582 lrc: 1/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a22179263c expref: -99 pid: 109937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572512.609051:0:109937:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572512.609075:0:109937:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109937:1875449863863168:192.168.202.132@tcp:103:lfs.0 00000100:00100000:0.0:1788572512.609189:0:109937:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827fcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.613009:0:109937:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109937:1875449863863168:192.168.202.132@tcp:103:lfs.0 00010000:00010000:2.0:1788572512.613041:0:109937:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc70205e00/0x58a604dcef352582 lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a22179263c expref: -99 pid: 109937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572512.613113:0:109937:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6733bb80 x1875449863862528/t4294983706(4294983706) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572512.613125:0:109937:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc69d47000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572512.613131:0:109937:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617f800 x1875449863863424/t0(0) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572512.613155:0:109937:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109937:1875449863863424:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572512.613237:0:109937:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fe40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572512.616406:0:109937:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job lfs:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109937:1875449863863424:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572512.616442:0:109937:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6733bb80 x1875449863862528/t4294983706(4294983706) o101->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572512.616507:0:109937:0:(file.c:479:ll_file_release()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665d1988), flags 154000 00010000:00010000:2.0:1788572512.616535:0:109937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc702f5000/0x58a604dcef35257b lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792627 expref: -99 pid: 109937 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572512.806457:0:109943:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:1.0:1788572512.806511:0:109943:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc69d47000 00010000:00010000:1.0:1788572512.806535:0:109943:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 6/6 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788572512.806642:0:109941:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc69d47000 00010000:00010000:0.0:1788572512.806672:0:109941:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc69d47000 00010000:00010000:0.0:1788572512.806690:0:109941:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/7 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1788572512.806762:0:109942:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc69d47000 00010000:00010000:0.0:1788572512.806768:0:109942:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/8 locks to regular blp list, 3 blwis in pool 00010000:00010000:1.0:1788572512.807747:0:109878:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572512.813325:0:109944:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572512.814401:0:109878:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400010000 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572512.814431:0:109878:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788572512.814444:0:109878:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1788572512.814505:0:109878:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572512.814513:0:109878:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572512.814535:0:109878:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109878:1875449863864064:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572512.814639:0:109878:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f540 going to sleep for 11 seconds 00010000:00010000:1.0:1788572512.816840:0:109944:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400010000 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572512.816868:0:109944:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572512.816880:0:109944:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788572512.816929:0:109944:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572512.816938:0:109944:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572512.816954:0:109944:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109944:1875449863864320:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572512.817008:0:109944:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572512.818396:0:109878:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109878:1875449863864064:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572512.818426:0:109878:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc69d47000 lock: 0000000007e4a0b9/0x58a604dcef352590 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a221792658 expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572512.818477:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572512.818481:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572512.818483:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572512.822000:0:109944:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109944:1875449863864320:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572512.822014:0:109944:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff92fc69d47000 lock: 0000000083e15edd/0x58a604dcef352589 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a22179264a expref: -99 pid: 109937 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572512.822047:0:109944:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572512.822050:0:109944:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572512.822051:0:109944:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.839914:0:109945:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572512.891021:0:71771:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041f:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572512.891039:0:71771:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572512.891069:0:71771:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041f:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572512.891084:0:71771:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572512.891090:0:71771:0:(lov_io.c:655:lov_io_mirror_init()) [0x20000041d:0x2:0x0]: flr state: 1, move mirror from 0 to 0, have retried: 0, mirror count: 2 00020000:00200000:2.0:1788572512.891094:0:71771:0:(lov_io.c:709:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:2.0:1788572512.891096:0:71771:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572512.891104:0:71771:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572513.863929:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572514.181294:0:109967:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665dc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572514.181318:0:109967:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:0.0:1788572514.181346:0:109967:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572514.181405:0:109967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 109967 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572514.181413:0:109967:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109967 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572514.181420:0:109967:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 109967 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572514.181440:0:109967:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job lsof:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109967:1875449863864448:192.168.202.132@tcp:101:lsof.0 00000100:00100000:0.0:1788572514.181604:0:109967:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5827f540 going to sleep for 11 seconds 00010000:00010000:1.0:1788572514.885941:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572515.337079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863864832:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572515.337103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863864704:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572515.338020:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e7bf4cc7-6786-4e88-b978-fb531985db08:2396:1875449863864576:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572515.338037:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863865088:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572515.338050:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2396:1875449863864960:192.168.202.132@tcp:400:kworker.0 00010000:00010000:1.0:1788572515.914040:0:109944:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572516.517481:0:109975:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665dc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572516.517496:0:109975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:0.0:1788572516.517520:0:109975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572516.517541:0:109975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572516.517559:0:109975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572516.517567:0:109975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572516.517578:0:109975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:0.0:1788572516.517583:0:109975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef352597 00000080:00010000:0.0:1788572516.517591:0:109975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352597 from it ffff9e0383737cb8 00010000:00010000:0.0:1788572516.517596:0:109975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572516.517606:0:109975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:0.0:1788572516.517905:0:109975:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665dc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572516.517910:0:109975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8),name=/ 00000002:00010000:0.0:1788572516.517918:0:109975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572516.517926:0:109975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572516.517935:0:109975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572516.517942:0:109975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572516.517947:0:109975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00010000:0.0:1788572516.517949:0:109975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665dc2c8) for lock 0x58a604dcef352597 00000080:00010000:0.0:1788572516.517953:0:109975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352597 from it ffff9e0383737cb8 00010000:00010000:0.0:1788572516.517957:0:109975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572516.519933:0:109975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572516.541354:0:109975:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665dc2c8) 00000080:00200000:1.0:1788572516.541392:0:109975:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc69d47000 (sb ffff92fc69d47000) 00010000:00010000:1.0:1788572516.541430:0:109975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541440:0:109975:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541448:0:109975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541455:0:109975:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541476:0:109975:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572516.541481:0:109975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541487:0:109975:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541492:0:109975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541501:0:109975:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541518:0:109975:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1788572516.541524:0:109975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541530:0:109975:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541536:0:109975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541541:0:109975:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541553:0:109975:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:1.0:1788572516.541565:0:109975:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:1.0:1788572516.541585:0:109877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788572516.541590:0:109877:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541598:0:109877:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366600 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572516.541601:0:109877:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541609:0:109877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572516.541619:0:109877:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572516.541625:0:109877:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572516.541627:0:109877:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:1.0:1788572516.541629:0:109877:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:1.0:1788572516.541662:0:109877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541668:0:109877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572516.541688:0:109877:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541695:0:109877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef35245c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179238e expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541709:0:109877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572516.541712:0:109877:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541717:0:109877:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3000 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572516.541746:0:109877:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541754:0:109877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572516.541761:0:109877:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572516.541769:0:109877:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572516.541772:0:109877:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:1.0:1788572516.541773:0:109877:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572516.541785:0:109877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541791:0:109877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572516.541797:0:109877:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541813:0:109877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3000/0x58a604dcef35248d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792395 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541825:0:109877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572516.541828:0:109877:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541833:0:109877:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367a00 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572516.541835:0:109877:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541842:0:109877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572516.541849:0:109877:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572516.541854:0:109877:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572516.541855:0:109877:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:1.0:1788572516.541857:0:109877:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:1.0:1788572516.541869:0:109877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541875:0:109877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572516.541880:0:109877:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541886:0:109877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70367a00/0x58a604dcef352494 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179239c expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.541895:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572516.541896:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572516.541898:0:109877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572516.541919:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e7bf4cc7-6786-4e88-b978-fb531985db08:2393:1875449863865216:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572516.542012:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e7bf4cc7-6786-4e88-b978-fb531985db08:2393:1875449863865472:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572516.542037:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572516.542042:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e7bf4cc7-6786-4e88-b978-fb531985db08:2393:1875449863865344:192.168.202.132@tcp:103:ldlm_bl.0 00020000:01000000:1.0:1788572516.542675:0:109975:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc50c38000 active: 1 00020000:01000000:1.0:1788572516.542681:0:109975:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc42c0b800 active: 1 00020000:01000000:1.0:1788572516.542684:0:109975:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:1.0:1788572516.542689:0:109975:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc69d47000 00020000:01000000:1.0:1788572516.542692:0:109975:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc69d47000: disconnecting target lustre-OST0001-osc-ffff92fc69d47000 00000100:00080000:1.0:1788572516.542704:0:109975:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-OST0001_UUID 00000100:00080000:1.0:1788572516.542820:0:109975:0:(import.c:65:import_set_state_nolock()) ffff92fc6714d000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:0.0:1788572516.542962:0:109878:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572516.549412:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e7bf4cc7-6786-4e88-b978-fb531985db08:2393:1875449863865216:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572516.566222:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e7bf4cc7-6786-4e88-b978-fb531985db08:2393:1875449863865344:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572516.566236:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e7bf4cc7-6786-4e88-b978-fb531985db08:2393:1875449863865472:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:1.0:1788572516.574471:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572516.574476:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc69d47000: committing for last_committed 4294968471 gen 2 00000100:00100000:1.0:1788572516.574481:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.574621:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3524be 00000020:00000080:1.0:1788572516.574640:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc42c0b800/LOV_OSC_UUID 00000020:01000000:1.0:1788572516.574691:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc69d47000 (flags='') 00000020:00000080:1.0:1788572516.574697:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572516.574706:0:109975:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc69d47000: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572516.574710:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc48cc8000) has no exports 00000020:00000080:0.0:1788572516.575441:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc42c0b800/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc69d47000 00000008:01000000:1.0:1788572516.576482:0:109975:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc69d47000: client import never connected 00000100:00080000:1.0:1788572516.576490:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572516.576494:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc69d47000: committing for last_committed 4294968471 gen 3 00000100:00100000:1.0:1788572516.576502:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.576714:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.576754:0:109975:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00020000:01000000:1.0:1788572516.576760:0:109975:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc69d47000 00020000:01000000:1.0:1788572516.576763:0:109975:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc69d47000: disconnecting target lustre-OST0000-osc-ffff92fc69d47000 00000100:00080000:1.0:1788572516.576782:0:109975:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-OST0000_UUID 00000100:00080000:1.0:1788572516.577470:0:109975:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7d000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572516.577603:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863865728:192.168.202.132@tcp:9:umount.0 00000100:00080000:2.0:1788572516.581590:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7d000 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572516.581619:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863865728:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572516.581935:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572516.581944:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc69d47000: committing for last_committed 4294968532 gen 2 00000100:00100000:1.0:1788572516.581955:0:109975:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617f100 x1875449863855232/t4294968533(4294968533) o10->lustre-OST0000-osc-ffff92fc69d47000@192.168.202.132@tcp:6/4 lens 440/432 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572516.581972:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.582109:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3524b7 00000020:00000080:1.0:1788572516.582131:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc50c38000/LOV_OSC_UUID 00000020:01000000:1.0:1788572516.582178:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc69d47000 (flags='') 00000020:00000080:1.0:1788572516.582185:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572516.582195:0:109975:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc69d47000: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572516.582199:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc48cc99c0) has no exports 00000008:01000000:1.0:1788572516.583131:0:109975:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc69d47000: client import never connected 00000100:00080000:1.0:1788572516.583136:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572516.583138:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc69d47000: committing for last_committed 4294968532 gen 3 00000100:00100000:1.0:1788572516.583143:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.583335:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.583343:0:109975:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:00000080:1.0:1788572516.583348:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3524b0 00000020:00000080:1.0:1788572516.583354:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc42b6d000/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000100:00080000:1.0:1788572516.583409:0:109975:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-MDT0000_UUID 00010000:00010000:1.0:1788572516.583539:0:109975:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572516.583590:0:109975:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572516.583598:0:109975:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572516.583613:0:109975:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329e40 going to sleep for 0 seconds 00000100:00100000:1.0:1788572516.583623:0:109975:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job umount:22e19d3b-9675-4547-8eb5-08aa5a0df26e:109975:1875449863865856:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572516.583632:0:109975:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572516.583639:0:109975:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572516.583642:0:109975:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc69d47000 lock: ffff92fc51bc2600/0x58a604dcef352597 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179266d expref: -99 pid: 109967 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00080000:1.0:1788572516.584774:0:109975:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7b000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572516.585132:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863865984:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572516.587442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5df7b000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572516.587475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863865984:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572516.587630:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572516.587635:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc69d47000: committing for last_committed 4294983690 gen 2 00000100:00100000:1.0:1788572516.587645:0:109975:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617f480 x1875449863854848/t4294983691(4294983691) o36->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572516.587672:0:109975:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67339180 x1875449863855872/t4294983692(4294983692) o36->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 488/456 e 0 to 0 dl 1788572527 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572516.587685:0:109975:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617df80 x1875449863857536/t4294983697(4294983697) o35->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572528 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572516.587694:0:109975:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6728a680 x1875449863865984/t4294983710(4294983710) o39->lustre-MDT0000-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572526 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572516.587707:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.590961:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3524a2 00000020:00000080:1.0:1788572516.590973:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc4871c800/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000100:00080000:1.0:1788572516.591113:0:109975:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22e19d3b-9675-4547-8eb5-08aa5a0df26e->lustre-MDT0001_UUID 00000100:00080000:1.0:1788572516.592107:0:109975:0:(import.c:65:import_set_state_nolock()) ffff92fc5df78000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572516.592166:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863866112:192.168.202.132@tcp:39:umount.0 00000020:00000080:0.0:1788572516.596464:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4871c800/22e19d3b-9675-4547-8eb5-08aa5a0df26e for lustre-MDT0000-mdc-ffff92fc69d47000 00000100:00080000:2.0:1788572516.600418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5df78000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572516.600495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22e19d3b-9675-4547-8eb5-08aa5a0df26e:2392:1875449863866112:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572516.600578:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572516.600582:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc69d47000: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572516.600589:0:109975:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6728a680 x1875449863866112/t4294976747(4294976747) o39->lustre-MDT0001-mdc-ffff92fc69d47000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572526 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572516.600608:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.602544:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3524a9 00000020:00000080:1.0:1788572516.602556:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc4871a800/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:00000080:1.0:1788572516.602595:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35249b 00000020:00000080:1.0:1788572516.602600:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc48718800/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:01000000:1.0:1788572516.602608:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc69d47000 (flags='') 00000020:00000080:1.0:1788572516.602617:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572516.602620:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4871a800/22e19d3b-9675-4547-8eb5-08aa5a0df26e for lustre-MDT0001-mdc-ffff92fc69d47000 00000020:00000080:1.0:1788572516.602623:0:109975:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc69d47000: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572516.602627:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc48d13380) has no exports 00000020:00000080:0.0:1788572516.602634:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48718800/22e19d3b-9675-4547-8eb5-08aa5a0df26e for lustre-clilmv-ffff92fc69d47000 00000020:00000080:1.0:1788572516.602681:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.602688:0:109975:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:00000080:1.0:1788572516.602691:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc5df78800/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:00000080:1.0:1788572516.602693:0:109975:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df78800/22e19d3b-9675-4547-8eb5-08aa5a0df26e for lustre-clilov-ffff92fc69d47000 00000020:01000000:1.0:1788572516.602696:0:109975:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc69d47000 (22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:1.0:1788572516.602913:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc69d47000 (flags='') 00000020:00000080:1.0:1788572516.602916:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572516.602918:0:109975:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc69d47000: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572516.602920:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc48d14d40) has no exports 00000020:00000080:1.0:1788572516.602952:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.602954:0:109975:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:00000080:1.0:1788572516.602956:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc5df7c800/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:00000080:1.0:1788572516.602958:0:109975:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df7c800/22e19d3b-9675-4547-8eb5-08aa5a0df26e for lustre-clilmv-ffff92fc69d47000 00000020:01000000:1.0:1788572516.602960:0:109975:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc69d47000 (22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:1.0:1788572516.603039:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc69d47000 (flags='') 00000020:00000080:1.0:1788572516.603042:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572516.603043:0:109975:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc69d47000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572516.603045:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc48d119c0) has no exports 00000008:01000000:1.0:1788572516.603179:0:109975:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc69d47000: client import never connected 00000100:00080000:1.0:1788572516.603182:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572516.603185:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc69d47000: committing for last_committed 4294983690 gen 3 00000100:00100000:1.0:1788572516.603189:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.605471:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.605482:0:109975:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000020:01000000:1.0:1788572516.605486:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc69d47000 (flags='') 00000020:00000080:1.0:1788572516.605488:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572516.605490:0:109975:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc69d47000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572516.605493:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc48d10000) has no exports 00000008:01000000:1.0:1788572516.606171:0:109975:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc69d47000: client import never connected 00000100:00080000:1.0:1788572516.606176:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572516.606181:0:109975:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc69d47000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572516.606187:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572516.608803:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.608815:0:109975:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc69d47000 (uuid 22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000080:02000400:1.0:1788572516.608819:0:109975:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572516.608820:0:109975:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1788572516.608852:0:109975:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc69d47000 00000100:00080000:1.0:1788572516.608859:0:109975:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import e7bf4cc7-6786-4e88-b978-fb531985db08->MGS 00010000:00010000:1.0:1788572516.608870:0:109975:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572516.608880:0:109975:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572516.608884:0:109975:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572516.608886:0:109975:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572516.608888:0:109975:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572516.608924:0:109975:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572516.608928:0:109975:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572516.608943:0:109975:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58329e40 going to sleep for 0 seconds 00000100:00100000:1.0:1788572516.608952:0:109975:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job umount:e7bf4cc7-6786-4e88-b978-fb531985db08:109975:1875449863866240:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572516.608959:0:109975:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572516.608964:0:109975:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572516.608967:0:109975:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366200/0x58a604dcef352455 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792387 expref: -99 pid: 109866 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572516.608985:0:109975:0:(import.c:65:import_set_state_nolock()) ffff92fc48719000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572516.609025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e7bf4cc7-6786-4e88-b978-fb531985db08:2392:1875449863866368:192.168.202.132@tcp:251:umount.0 00000020:00000080:0.0:1788572516.609126:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5df78000 for lustre-MDT0001-mdc-ffff92fc69d47000 00000020:00000080:0.0:1788572516.609133:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc5df7e800/22e19d3b-9675-4547-8eb5-08aa5a0df26e 00000020:00000080:0.0:1788572516.609137:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5df7e800/22e19d3b-9675-4547-8eb5-08aa5a0df26e for lustre-MDT0001-mdc-ffff92fc69d47000 00000020:01000000:0.0:1788572516.609142:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc69d47000 (22e19d3b-9675-4547-8eb5-08aa5a0df26e) 00000100:00080000:2.0:1788572516.612039:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48719000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:2.0:1788572516.612054:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572516.612057:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572516.612059:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572516.612074:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e7bf4cc7-6786-4e88-b978-fb531985db08:2392:1875449863866368:192.168.202.132@tcp:251:umount.0 00000100:00080000:1.0:1788572516.616036:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572516.616119:0:109975:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572516.616124:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572516.616127:0:109975:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572516.616142:0:109975:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35244e 00000020:00000080:1.0:1788572516.616157:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc58336800/e7bf4cc7-6786-4e88-b978-fb531985db08 00000020:01000000:1.0:1788572516.616175:0:109975:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:1.0:1788572516.616400:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572516.616413:0:109975:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572516.616419:0:109975:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc51cd99c0) has no exports 00000020:00000080:1.0:1788572516.616769:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58336800/e7bf4cc7-6786-4e88-b978-fb531985db08 for MGC192.168.202.132@tcp 10000000:01000000:2.0:1788572516.617422:0:109881:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:1.0:1788572516.617445:0:109975:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:1.0:1788572516.617449:0:109975:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572516.617452:0:109975:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572516.617455:0:109975:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572516.617457:0:109975:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572516.617489:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc48719000 for MGC192.168.202.132@tcp 00000020:00000080:1.0:1788572516.617535:0:109975:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572516.617543:0:109975:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid e7bf4cc7-6786-4e88-b978-fb531985db08) 00000020:00000080:1.0:1788572516.617547:0:109975:0:(genops.c:885:class_export_put()) final put ffff92fc69d42800/e7bf4cc7-6786-4e88-b978-fb531985db08 00000020:00000080:1.0:1788572516.617550:0:109975:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d42800/e7bf4cc7-6786-4e88-b978-fb531985db08 for MGC192.168.202.132@tcp 00000020:01000000:1.0:1788572516.617554:0:109975:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (e7bf4cc7-6786-4e88-b978-fb531985db08) 00010000:00010000:1.0:1788572516.617810:0:109944:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572516.617920:0:109877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:2.0:1788572516.618446:0:109878:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00010000:00010000:2.0:1788572516.629898:0:109945:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-4 blwis in pool) 00000100:00100000:1.0:1788572516.630312:0:109868:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc51035800:109868] 1 exiting: rc = 0 00000100:00100000:0.0:1788572516.630540:0:109867:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc49952e00:109867] 0 exiting: rc = 1 00000100:00100000:2.0:1788572516.631139:0:109869:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc49952c00:109869] 0 exiting: rc = 1 00000100:00100000:2.0:1788572516.631195:0:109870:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff92fc51034c00:109870] 1 exiting: rc = 0 00000100:00100000:1.0:1788572516.631402:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.631412:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.631632:0:109871:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc60a84000:109871] 0 exiting: rc = 1 00000100:00100000:1.0:1788572516.631679:0:109872:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc60a85200:109872] 1 exiting: rc = 0 00000100:00100000:1.0:1788572516.631752:0:109873:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc4a7c1400:109873] 2 exiting: rc = 0 00000100:00100000:2.0:1788572516.632104:0:109874:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc53eab200:109874] 0 exiting: rc = 1 00000100:00100000:2.0:1788572516.632808:0:109876:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc49ba8000:109876] 2 exiting: rc = 0 00000100:00100000:2.0:1788572516.632875:0:109875:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc53eaa600:109875] 1 exiting: rc = 0 00000100:00100000:1.0:1788572516.633111:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633119:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633124:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633130:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633136:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633141:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633146:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633151:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633156:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633162:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633168:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633173:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633179:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633184:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633189:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633195:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633200:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633205:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633210:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633215:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633220:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633225:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633231:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633238:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633243:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633248:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633253:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633258:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633263:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633268:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633274:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633279:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633284:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633289:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633296:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633336:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633344:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633349:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633355:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633361:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633366:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633370:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633375:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633380:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633387:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633391:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633395:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633399:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633406:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633412:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633417:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633422:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633427:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633433:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633438:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633442:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633447:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633453:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633458:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633463:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633468:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633472:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633477:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633482:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633489:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633494:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633499:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633504:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633509:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633513:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633519:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633524:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633528:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633534:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633539:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633544:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633550:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633555:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633561:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633566:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633571:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633576:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633581:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633585:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633590:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633595:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633600:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633604:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633609:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633623:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633631:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633636:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633642:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633647:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633652:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633657:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633662:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633667:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633671:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633676:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633681:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633686:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633691:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633696:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633701:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633706:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633711:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633716:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633738:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633744:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633749:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633754:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633759:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633764:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633770:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633775:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633780:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633784:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633789:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633801:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633807:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633812:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633817:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633823:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633827:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633832:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633837:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572516.633841:0:109975:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:1.0:1788572516.635646:0:109975:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc69d47000 1 00000020:01000004:1.0:1788572516.635653:0:109975:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc69d47800 00000100:00100000:1.0:1788572517.238222:0:109988:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:0.0:1788572517.238361:0:109988:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:0.0:1788572517.238945:0:109988:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:0.0:1788572517.238952:0:109988:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:0.0:1788572517.239426:0:109988:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572517.239432:0:109988:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:0.0:1788572517.241457:0:109988:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572517.245179:0:109988:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572517.245537:0:109988:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:0.0:1788572517.245543:0:109988:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:1.0:1788572517.248417:0:109988:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:1.0:1788572517.248423:0:109988:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:2.0:1788572517.249181:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572517.249184:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572517.249186:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788572517.249548:0:109988:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5824c800@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572517.249933:0:109988:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:1.0:1788572517.249941:0:109988:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:2.0:1788572517.254179:0:110002:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:0.0:1788572517.256472:0:109988:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid 9d8e3785-0a8f-445a-94cc-8fe7d50612c2) 00000020:00000080:0.0:1788572517.256531:0:109988:0:(genops.c:1385:class_connect()) connect: client 9d8e3785-0a8f-445a-94cc-8fe7d50612c2, cookie 0x58a604dcef3525a5 00000100:00080000:0.0:1788572517.256548:0:109988:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572517.257386:0:109988:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572517.257428:0:109988:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449863866496/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572517.257499:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:2392:1875449863866496:192.168.202.132@tcp:250:mount.lustre.0 00000100:00080000:0.0:1788572517.258133:0:109988:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 9d8e3785-0a8f-445a-94cc-8fe7d50612c2->MGS 00000080:00200000:0.0:1788572517.258155:0:109988:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc58389800 (sb ffff92fc58389800) 00000080:01000000:0.0:1788572517.258239:0:109988:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock,test_dummy_encryption 00000080:00000004:0.0:1788572517.258242:0:109988:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788572517.258246:0:109988:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:00000004:0.0:1788572517.258248:0:109988:0:(llite_lib.c:1242:ll_options()) next opt=test_dummy_encryption 00000080:01000000:0.0:1788572517.258257:0:109988:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 495b6acd-313a-4a79-a435-ab1ce65f91fe 10000000:01000000:0.0:1788572517.258998:0:109988:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788572517.259002:0:109988:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc58389800 10000000:01000000:0.0:1788572517.259007:0:109988:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc51cd8000 10000000:01000000:0.0:1788572517.259012:0:109988:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572517.259019:0:109988:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc51cd8000 from 1 10000000:01000000:0.0:1788572517.259022:0:109988:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572517.259058:0:109988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572517.259069:0:109988:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572517.259080:0:109988:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572517.259103:0:109988:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67048000 x1875449863866624/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572517.259113:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc43d9b900 going to sleep for 0 seconds 00000100:00100000:2.0:1788572517.268670:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572517.268688:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:2.0:1788572517.268698:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:2.0:1788572517.268705:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:2.0:1788572517.268710:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:2.0:1788572517.268714:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572517.277443:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:2.0:1788572517.277467:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000ebc9553a): req@ffff92fc67048000 x1875449863866624/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572517.277489:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:2392:1875449863866496:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572517.286906:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc43d9b900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.295094:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863866624:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572517.295124:0:109988:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179267b expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.295135:0:109988:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179267b expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572517.295205:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863866752:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572517.295384:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.301138:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863866752:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:2.0:1788572517.301191:0:109988:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572517.301212:0:109988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179267b expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.301225:0:109988:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3c00/0x58a604dcef3525ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179267b expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572517.301235:0:109988:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc58389800 10000000:01000000:2.0:1788572517.301242:0:109988:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:2.0:1788572517.301247:0:109988:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc58389800 10000000:01000000:2.0:1788572517.301249:0:109988:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572517.301254:0:109988:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc58389800 10000000:01000000:2.0:1788572517.301256:0:109988:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:2.0:1788572517.301260:0:109988:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc58389800 from 1 10000000:01000000:2.0:1788572517.301263:0:109988:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788572517.301314:0:109988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.301323:0:109988:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.301333:0:109988:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572517.301353:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863866880:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572517.301432:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.304137:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863866880:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572517.304164:0:109988:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.304176:0:109988:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572517.304247:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867008:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572517.304381:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.316663:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867008:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572517.316762:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867136:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572517.316862:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.320307:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867136:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572517.320896:0:110007:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110007:1875449863867264:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572517.321009:0:110007:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58e69d80 going to sleep for 11 seconds 00000020:00000080:2.0:1788572517.332944:0:110007:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:2.0:1788572517.332961:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc58389800 00000020:00000080:2.0:1788572517.332965:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572517.332993:0:110007:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788572517.333032:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572517.333048:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.333050:0:110007:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788572517.333054:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572517.333056:0:110007:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:2.0:1788572517.333058:0:110007:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572517.333060:0:110007:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572517.333061:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572517.333064:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.333065:0:110007:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572517.333067:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572517.333072:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.333073:0:110007:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:2.0:1788572517.333076:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572517.333081:0:110007:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572517.333108:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc58389800 00000020:00000080:2.0:1788572517.333110:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572517.333121:0:110007:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc58389800 (ffff92fc48cccd40) 00000020:00000080:2.0:1788572517.333137:0:110007:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:2.0:1788572517.333142:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc58389800 00000020:00000080:2.0:1788572517.333145:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572517.333214:0:110007:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc6714a800@lustre-MDT0001-mdc-ffff92fc58389800: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572517.334845:0:110007:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc58389800-2 is initialized 00000040:01000000:2.0:1788572517.335050:0:110007:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc58389800 ctxt 13 is initialized 00000020:00000080:2.0:1788572517.335205:0:110007:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:2.0:1788572517.335219:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc58389800 00000020:00000080:2.0:1788572517.335223:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572517.335244:0:110007:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:2.0:1788572517.335276:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572517.335280:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.335281:0:110007:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:2.0:1788572517.335284:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572517.335287:0:110007:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:2.0:1788572517.335288:0:110007:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572517.335290:0:110007:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572517.335291:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572517.335294:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.335307:0:110007:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572517.335311:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572517.335316:0:110007:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572517.335319:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.335321:0:110007:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788572517.335323:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572517.335329:0:110007:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572517.335353:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc58389800 00000020:00000080:2.0:1788572517.335355:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572517.335365:0:110007:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc58389800 (ffff92fc48ccb380) 00000020:00000080:2.0:1788572517.335377:0:110007:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788572517.335382:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc58389800 00000020:00000080:2.0:1788572517.335384:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572517.335440:0:110007:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990a000@lustre-OST0000-osc-ffff92fc58389800: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572517.335622:0:110007:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc58389800-3 is initialized 00000020:00000080:2.0:1788572517.335835:0:110007:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:2.0:1788572517.335842:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc58389800 00000020:00000080:2.0:1788572517.335845:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572517.335857:0:110007:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788572517.335863:0:110007:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc58389800: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788572517.335868:0:110007:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788572517.335878:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572517.335882:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.335884:0:110007:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788572517.335886:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572517.335888:0:110007:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:2.0:1788572517.335889:0:110007:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572517.335891:0:110007:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572517.335892:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572517.335895:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.335896:0:110007:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572517.335898:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572517.335901:0:110007:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572517.335904:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.335905:0:110007:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788572517.335908:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572517.335911:0:110007:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572517.335923:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc58389800 00000020:00000080:2.0:1788572517.335926:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572517.335932:0:110007:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc58389800 (ffff92fc48cc99c0) 00000020:00000080:2.0:1788572517.335940:0:110007:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:2.0:1788572517.335945:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc58389800 00000020:00000080:2.0:1788572517.335947:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572517.335991:0:110007:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990a800@lustre-OST0001-osc-ffff92fc58389800: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572517.336166:0:110007:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc58389800-4 is initialized 00000020:00000080:2.0:1788572517.339321:0:110007:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:2.0:1788572517.339339:0:110007:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc58389800 00000020:00000080:2.0:1788572517.339344:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572517.339369:0:110007:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1788572517.339377:0:110007:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc58389800: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1788572517.339382:0:110007:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1788572517.339392:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572517.339398:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.339399:0:110007:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1788572517.339403:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572517.339407:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.339408:0:110007:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:2.0:1788572517.339411:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1788572517.339412:0:110007:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1788572517.339414:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572517.339416:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.339417:0:110007:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:2.0:1788572517.339460:0:110007:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110007:1875449863867392:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572517.339580:0:110007:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5540 going to sleep for 11 seconds 00000100:00100000:1.0:1788572517.354651:0:110007:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110007:1875449863867392:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572517.354696:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572517.354707:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572517.354710:0:110007:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788572517.354741:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788572517.354782:0:110007:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110007:1875449863867520:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572517.354869:0:110007:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d75b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572517.360742:0:110007:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110007:1875449863867520:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572517.360798:0:110007:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572517.360806:0:110007:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572517.360809:0:110007:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:0.0:1788572517.360813:0:110007:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:2.0:1788572517.361282:0:109988:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:2.0:1788572517.361323:0:109988:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572517.361340:0:109988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.361357:0:109988:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572517.361366:0:109988:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc58389800 from 1 10000000:01000000:2.0:1788572517.361368:0:109988:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572517.361401:0:109988:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.361409:0:109988:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.361419:0:109988:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572517.361442:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867648:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572517.361527:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ab40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.375355:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867648:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572517.375388:0:109988:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.375401:0:109988:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572517.376240:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863867776:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572517.376451:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ab40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572517.394281:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863868032:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572517.394380:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc43d9b540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572517.396823:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863868032:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572517.396886:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863868160:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572517.396968:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc43d9b540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572517.400511:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:109988:1875449863868160:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572517.401106:0:110013:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110013:1875449863868288:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572517.401212:0:110013:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d75e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.419497:0:110013:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:110013:1875449863868416:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572517.419558:0:110013:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572517.419572:0:110013:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572517.419575:0:110013:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:2.0:1788572517.419580:0:110013:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:0.0:1788572517.423436:0:109988:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:0.0:1788572517.423474:0:109988:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572517.424531:0:109988:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef3525eb lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792690 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572517.424547:0:109988:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef3525eb lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792690 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:0.0:1788572517.424575:0:109988:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572517.424603:0:109988:0:(genops.c:1385:class_connect()) connect: client 495b6acd-313a-4a79-a435-ab1ce65f91fe, cookie 0x58a604dcef3525f2 00800000:01000000:0.0:1788572517.424631:0:109988:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 495b6acd-313a-4a79-a435-ab1ce65f91fe to lustre-clilmv-ffff92fc58389800 00800000:01000000:0.0:1788572517.424640:0:109988:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc58389800(495b6acd-313a-4a79-a435-ab1ce65f91fe) - lustre-MDT0000_UUID, 495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:0.0:1788572517.424648:0:109988:0:(genops.c:1385:class_connect()) connect: client 495b6acd-313a-4a79-a435-ab1ce65f91fe, cookie 0x58a604dcef3525f9 00000100:00080000:0.0:1788572517.424659:0:109988:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc58389800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572517.424671:0:109988:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc58389800: import ffff92fc5106b000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572517.424690:0:109988:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572517.424735:0:109988:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449863868544/t0(0) o38->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572517.424756:0:109988:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-MDT0000_UUID 00800000:01000000:0.0:1788572517.424843:0:109988:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc58389800(495b6acd-313a-4a79-a435-ab1ce65f91fe) successfully (5) 00800000:00000004:0.0:1788572517.424848:0:109988:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc58389800: age 3273, max_age 0 00800000:01000000:0.0:1788572517.424875:0:109988:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc58389800(495b6acd-313a-4a79-a435-ab1ce65f91fe) - lustre-MDT0001_UUID, 495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:0.0:1788572517.424883:0:109988:0:(genops.c:1385:class_connect()) connect: client 495b6acd-313a-4a79-a435-ab1ce65f91fe, cookie 0x58a604dcef352600 00000100:00080000:0.0:1788572517.424888:0:109988:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc58389800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572517.424893:0:109988:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc58389800: import ffff92fc6714a800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572517.424898:0:109988:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572517.424908:0:109988:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863868800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572517.424916:0:109988:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-MDT0001_UUID 00800000:01000000:0.0:1788572517.424951:0:109988:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc58389800(495b6acd-313a-4a79-a435-ab1ce65f91fe) successfully (5) 00800000:00000004:0.0:1788572517.424954:0:109988:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc58389800: age 3273, max_age 0 00000080:00000004:0.0:1788572517.424969:0:109988:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc58389800: age 3273, max_age 4272 00800000:00000004:0.0:1788572517.424973:0:109988:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc58389800: age 3273, max_age 4272 00000100:00080000:0.0:1788572517.424992:0:109988:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67049880 x1875449863869056/t0(0) o41->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572517.425001:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 0 seconds 00000100:00080000:0.0:1788572517.425037:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67048380 x1875449863868928/t0(0) o41->lustre-MDT0001-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572517.425049:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:0.0:1788572517.425054:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704aa00 x1875449863868672/t0(0) o41->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572517.425202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863868544:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572517.425431:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863868800:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572517.447013:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc58389800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572517.447031:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc58389800: connect to target with instance 1 00000100:00080000:2.0:1788572517.447041:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc58389800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572517.447055:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1788572517.447059:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5106b000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:0.0:1788572517.448244:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000005e58fe07): req@ffff92fc67049880 x1875449863869056/t0(0) o41->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572517.448269:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002939d3c5): req@ffff92fc6704aa00 x1875449863868672/t0(0) o41->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572517.448284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863868544:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788572517.448333:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc58389800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572517.448342:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc58389800: connect to target with instance 1 00000100:00080000:0.0:1788572517.448351:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc58389800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:0.0:1788572517.448361:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:0.0:1788572517.448365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc6714a800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:0.0:1788572517.448623:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572517.448882:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863868800:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788572517.455052:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:495b6acd-313a-4a79-a435-ab1ce65f91fe:0:1875449863868672:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572517.461933:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:495b6acd-313a-4a79-a435-ab1ce65f91fe:0:1875449863868928:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572517.463995:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869056:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:1.0:1788572517.464481:0:109988:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc58389800: update ffff92fc51cdd198 cache blocks 317012/353529 objects 1023701/1024000 00000080:00000004:1.0:1788572517.464495:0:109988:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc58389800: update ffff92fc51cdb7d8 cache blocks 317012/353529 objects 1023701/1024000 00000100:00100000:0.0:1788572517.470225:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869184:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:0.0:1788572517.470323:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572517.476104:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869184:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:1.0:1788572517.476136:0:109988:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:1.0:1788572517.476150:0:109988:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:1.0:1788572517.476154:0:109988:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:1.0:1788572517.476165:0:109988:0:(genops.c:1385:class_connect()) connect: client 495b6acd-313a-4a79-a435-ab1ce65f91fe, cookie 0x58a604dcef352607 00000020:00000080:1.0:1788572517.476182:0:109988:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef35260e 00000100:00080000:1.0:1788572517.476191:0:109988:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc58389800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572517.476198:0:109988:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc58389800: import ffff92fc4990a000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572517.476212:0:109988:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572517.476239:0:109988:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449863869312/t0(0) o8->lustre-OST0000-osc-ffff92fc58389800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572517.476267:0:109988:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-OST0000_UUID 00020000:01000000:1.0:1788572517.476279:0:109988:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc58389800) active 00000020:00000080:1.0:1788572517.476313:0:109988:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352615 00000100:00080000:1.0:1788572517.476318:0:109988:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc58389800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572517.476324:0:109988:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc58389800: import ffff92fc4990a800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572517.476330:0:109988:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572517.476341:0:109988:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449863869440/t0(0) o8->lustre-OST0001-osc-ffff92fc58389800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572517.476351:0:109988:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-OST0001_UUID 00020000:01000000:1.0:1788572517.476356:0:109988:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc58389800) active 00000080:02000400:1.0:1788572517.476364:0:109988:0:(llite_lib.c:801:client_common_fill_super()) Test dummy encryption mode enabled 00000100:00100000:1.0:1788572517.476398:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869568:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:1.0:1788572517.476468:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d75c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572517.476950:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863869312:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572517.477006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863869440:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572517.482523:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc58389800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572517.482545:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc58389800: connect to target with instance 1 00000100:00080000:2.0:1788572517.482557:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc58389800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572517.482573:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572517.482577:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572517.482585:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1788572517.482590:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572517.482594:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572517.482611:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863869312:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572517.487179:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc58389800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572517.487197:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc58389800: connect to target with instance 1 00000100:00080000:2.0:1788572517.487207:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc58389800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572517.487220:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788572517.487223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572517.487232:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:2.0:1788572517.487237:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572517.487242:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572517.487257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863869440:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572517.488268:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869568:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572517.488349:0:109988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869696:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572517.488470:0:109988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.502282:0:109988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:495b6acd-313a-4a79-a435-ab1ce65f91fe:109988:1875449863869696:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:2.0:1788572517.502500:0:109988:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00200000:2.0:1788572517.502509:0:109988:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:2.0:1788572517.502575:0:109988:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:2.0:1788572517.502579:0:109988:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00000080:00200000:1.0:1788572517.835651:0:110031:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:1.0:1788572517.835696:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572517.835794:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572517.835806:0:110031:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572517.835819:0:110031:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572517.835846:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863869824:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572517.835934:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d75e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.841130:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863869824:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572517.841162:0:110031:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841176:0:110031:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841188:0:110031:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841210:0:110031:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841231:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841241:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841263:0:110031:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572517.841276:0:110031:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67288380 x1875449863869824/t0(0) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572517.841290:0:110031:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841334:0:110031:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc67270000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572517.841342:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572517.841371:0:110031:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572517.841376:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:2.0:1788572517.841381:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:2.0:1788572517.841384:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e03837cfb58 00010000:00010000:2.0:1788572517.841388:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841397:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572517.841420:0:110031:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572517.841442:0:110031:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e1988), flags=25 00000080:00200000:2.0:1788572517.841447:0:110031:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e1988), intent=0 00000002:00010000:2.0:1788572517.841472:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572517.841519:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841527:0:110031:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.841537:0:110031:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572517.841567:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863869952:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572517.841649:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572517.846613:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863869952:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572517.846643:0:110031:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846670:0:110031:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846677:0:110031:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846688:0:110031:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846698:0:110031:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846710:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846764:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846772:0:110031:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572517.846786:0:110031:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617c380 x1875449863869952/t0(0) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572533 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572517.846798:0:110031:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846809:0:110031:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc67270000 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:2.0:1788572517.846814:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572517.846849:0:110031:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148) 00000080:00200000:2.0:1788572517.846857:0:110031:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572517.846872:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e1148) for lock 0x58a604dcef352623 00000080:00010000:2.0:1788572517.846884:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e1148) 00000080:00010000:2.0:1788572517.846888:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352623 from it ffff9e03837cfa28 00010000:00010000:2.0:1788572517.846891:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572517.846901:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572517.846932:0:110031:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788572517.846949:0:110031:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f59b.sanity-sec, dir=[0x20000041d:0x1:0x0](ffff92fc665e1148), flags=9 00000080:00200000:2.0:1788572517.846954:0:110031:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f59b.sanity-sec, dir=[0x20000041d:0x1:0x0](ffff92fc665e1148), intent=getattr 00000100:00100000:0.0:1788572518.016982:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863870080:192.168.202.132@tcp:900:lfs.0 00000100:00100000:0.0:1788572518.017072:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572518.023699:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863870080:192.168.202.132@tcp:900:lfs.0 00000002:00010000:0.0:1788572518.023794:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572518.023872:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.023883:0:110031:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.023894:0:110031:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572518.023917:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863870208:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572518.023983:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572518.033576:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863870208:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572518.033604:0:110031:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033626:0:110031:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033635:0:110031:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033648:0:110031:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033663:0:110031:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033683:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033691:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033699:0:110031:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572518.033711:0:110031:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704a680 x1875449863870208/t0(0) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 592/856 e 0 to 0 dl 1788572534 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572518.033766:0:110031:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572518.033781:0:110031:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.033891:0:110031:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc67270000 type 13 mode 4 res 8589935645/2 (0 0) 00000080:00200000:0.0:1788572518.033900:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572518.033948:0:110031:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00200000:0.0:1788572518.034041:0:110031:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00200000:0.0:1788572518.034060:0:110031:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:0.0:1788572518.034071:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:0.0:1788572518.034084:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:0.0:1788572518.034088:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfb20 00010000:00010000:0.0:1788572518.034091:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.034102:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572518.034929:0:110031:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572518.034942:0:110031:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:0.0:1788572518.034986:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.035000:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572518.035012:0:110031:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572518.035020:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.035025:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:0.0:1788572518.035028:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:0.0:1788572518.035033:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfcb8 00010000:00010000:0.0:1788572518.035037:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572518.035045:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000080:00200000:0.0:1788572518.035112:0:110031:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:0.0:1788572518.035130:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.035138:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572518.035152:0:110031:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572518.035159:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.035163:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:0.0:1788572518.035165:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:0.0:1788572518.035169:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e03837cfbf8 00010000:00010000:0.0:1788572518.035172:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572518.036780:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572518.036809:0:110031:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572518.036823:0:110031:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:0.0:1788572518.036831:0:110031:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148), inode mode 41ed mask 1 00000080:00200000:0.0:1788572518.036835:0:110031:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f59b.sanity-sec, flags=265 00000080:00200000:0.0:1788572518.036840:0:110031:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), inode mode 81a4 mask 44 00000080:00200008:0.0:1788572518.036850:0:110031:0:(file.c:1090:ll_file_open()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 154400 00000002:00010000:0.0:1788572518.036956:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0100240114401 00010000:00010000:0.0:1788572518.037019:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.037028:0:110031:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572518.037040:0:110031:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572518.037055:0:110031:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704a680 x1875449863870464/t0(0) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572518.037063:0:110031:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110031 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572518.037083:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863870464:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572518.037280:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723ff00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572518.045425:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863870464:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572518.045456:0:110031:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045476:0:110031:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045485:0:110031:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045496:0:110031:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045508:0:110031:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045521:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045529:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045536:0:110031:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572518.045548:0:110031:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc6704a680 x1875449863870464/t4294983711(4294983711) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572518.045577:0:110031:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6704a680 x1875449863870464/t4294983711(4294983711) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572518.045591:0:110031:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572518.045604:0:110031:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045617:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572518.045630:0:110031:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572518.045638:0:110031:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045645:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045652:0:110031:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.045679:0:110031:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1788572518.045686:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00010000:00010000:2.0:1788572518.046137:0:110000:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572518.046145:0:110000:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046153:0:110000:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2e00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572518.046158:0:110000:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046167:0:110000:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046176:0:110000:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 3/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046193:0:110000:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572518.046231:0:110000:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 2/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046238:0:110000:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572518.046260:0:110000:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046267:0:110000:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc2e00/0x58a604dcef352631 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792700 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.046281:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572518.046284:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572518.046285:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572518.046327:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2396:1875449863870592:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572518.052277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2396:1875449863870592:192.168.202.132@tcp:103:ldlm_bl.0 00000080:00200080:2.0:1788572518.149906:0:110031:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=1 00000080:00200008:2.0:1788572518.149927:0:110031:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572518.149936:0:110031:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:2.0:1788572518.149986:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572518.150008:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572518.150023:0:110031:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572518.150032:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572518.150039:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572518.150042:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:2.0:1788572518.150050:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfcd8 00010000:00010000:2.0:1788572518.150053:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572518.150062:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200008:2.0:1788572518.150093:0:110031:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 0, count: 9000 00000080:00200000:2.0:1788572518.150118:0:110031:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 0, bytes: 9000 00000080:00200000:2.0:1788572518.150127:0:110031:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 1 00000020:00200000:2.0:1788572518.150136:0:110031:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d7480875[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572518.150138:0:110031:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 9000) 2048 0 00000080:00200000:2.0:1788572518.150144:0:110031:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572518.150158:0:110031:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:2.0:1788572518.150167:0:110031:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 9000) 9000, 9000 00020000:00200000:2.0:1788572518.150170:0:110031:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572518.150212:0:110031:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 2] 00000008:00010000:2.0:1788572518.150226:0:110031:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:2.0:1788572518.150230:0:110031:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550730, osc lock ffff92fc58370be0, flags 0x40000 00000080:00200000:2.0:1788572518.150239:0:110031:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [0, 9000) 00000001:00010000:2.0:1788572518.150252:0:110031:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572518.150256:0:110031:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572518.150258:0:110031:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550930, osc lock ffff92fc58371690, flags 0x400001002 00010000:00010000:2.0:1788572518.151804:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.151822:0:110031:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572518.151870:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788572518.151886:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572518.151909:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2396:1875449863870976:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572518.156611:0:2396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.156638:0:2396:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.156652:0:2396:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.156674:0:2396:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572518.156690:0:2396:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572518.156703:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572518.156932:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.156942:0:2396:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.156949:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.156957:0:2396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572518.156978:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2396:1875449863870976:192.168.202.132@tcp:101:lfs.0 00000080:00200000:1.0:1788572518.163743:0:110031:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:1.0:1788572518.163757:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc704772e0 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:1.0:1788572518.163762:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc704772e0 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.163768:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550930 (ffff92fc6721ad88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.163774:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc704772e0 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572518.163777:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550930 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572518.163811:0:110031:0:(vvp_io.c:893:vvp_io_read_start()) count:9000 iocb pos:0 00000080:00200000:1.0:1788572518.163817:0:110031:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), size=9000 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel, unaligned 00000020:00200000:1.0:1788572518.163884:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:1.0:1788572518.163888:0:110031:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 9000 count 9000 file_offset 9000 00000020:00010000:1.0:1788572518.163895:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e35d8 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:1.0:1788572518.163897:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e35d8 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.163900:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550730 (ffff92fc6721ad88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.163902:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e35d8 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572518.163904:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550730 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572518.163910:0:110031:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000020:00200000:1.0:1788572518.163917:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1788572518.163971:0:103957:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc48227000 00000008:00100000:1.0:1788572518.164019:0:103957:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc6728aa00 - object 0x2c0000401:537 offset 0<>12288 00000100:00100000:0.0:1788572518.166143:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2394:1875449863871104:192.168.202.132@tcp:3:lfs.0 00000008:00010000:1.0:1788572518.176473:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc48227000 00000020:00200000:1.0:1788572518.176489:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:1.0:1788572518.176492:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:1.0:1788572518.176494:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1788572518.176500:0:2394:0:(cl_io.c:1302:cl_sub_dio_end()) finishing unaligned dio read aio->cda_bytes 9000 00000020:00200000:1.0:1788572518.176510:0:2394:0:(cl_io.c:1693:__ll_dio_user_copy()) count 9000, offset 0, pos 0, cdp_page_count 3 00000020:00200000:1.0:1788572518.176527:0:2394:0:(cl_io.c:1693:__ll_dio_user_copy()) count 4904, offset 0, pos 4096, cdp_page_count 3 00000020:00200000:1.0:1788572518.176531:0:2394:0:(cl_io.c:1693:__ll_dio_user_copy()) count 808, offset 0, pos 8192, cdp_page_count 3 00000020:00200000:1.0:1788572518.176541:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788572518.176575:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc48227000 00000100:00100000:1.0:1788572518.176606:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2394:1875449863871104:192.168.202.132@tcp:3:lfs.0 00000080:00200000:2.0:1788572518.177465:0:110031:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:2.0:1788572518.177480:0:110031:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572518.177486:0:110031:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000020:00200000:2.0:1788572518.177490:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572518.177500:0:110031:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200080:2.0:1788572518.177520:0:110031:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200080:2.0:1788572518.177586:0:110031:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=2 00000080:00200008:2.0:1788572518.177596:0:110031:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572518.177603:0:110031:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:2.0:1788572518.177648:0:110031:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572518.177666:0:110031:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572518.177681:0:110031:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572518.177689:0:110031:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572518.177696:0:110031:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572518.177700:0:110031:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:2.0:1788572518.177707:0:110031:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfcd8 00010000:00010000:2.0:1788572518.177711:0:110031:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572518.177760:0:110031:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00000080:00200008:2.0:1788572518.177863:0:110031:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 0, count: 9000 00000080:00200000:2.0:1788572518.177873:0:110031:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 0, bytes: 9000 00000080:00200000:2.0:1788572518.177972:0:110031:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 2 00000020:00200000:2.0:1788572518.177981:0:110031:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d7480875[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572518.177983:0:110031:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 9000) 2048 0 00000080:00200000:2.0:1788572518.177988:0:110031:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:2.0:1788572518.178002:0:110031:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:2.0:1788572518.178010:0:110031:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 9000) 9000, 9000 00020000:00200000:2.0:1788572518.178014:0:110031:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:2.0:1788572518.178029:0:110031:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 2] 00000008:00010000:2.0:1788572518.178154:0:110031:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 11 00000008:00010000:2.0:1788572518.178160:0:110031:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550630, osc lock ffff92fc58370ab0, flags 0x40000 00000080:00200000:2.0:1788572518.178167:0:110031:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [0, 9000) 00000080:00200000:2.0:1788572518.178174:0:110031:0:(vvp_io.c:194:vvp_prep_size()) [0x20000041d:0x2:0x0] updating i_size 9000 00000080:00200000:2.0:1788572518.178181:0:110031:0:(vvp_io.c:893:vvp_io_read_start()) count:9000 iocb pos:0 00000080:00200000:2.0:1788572518.178255:0:110031:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), size=9000 (max 536870912), offset=0=0x0, pages 3 (max 131072), lockless, parallel, unaligned 00000020:00200000:2.0:1788572518.178321:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:2.0:1788572518.178324:0:110031:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 9000 count 9000 file_offset 9000 00000020:00010000:2.0:1788572518.178331:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e35d8 (ffff92fc6721a368/0) at cl_lock_release():245 00000020:00010000:2.0:1788572518.178334:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e35d8 (ffff92fc6721a368/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572518.178336:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550630 (ffff92fc6721ad88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572518.178339:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e35d8 (ffff92fc6721a368/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572518.178342:0:110031:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550630 (ffff92fc6721a368/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572518.178349:0:110031:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000020:00200000:2.0:1788572518.178355:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572518.178398:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc58ca8480 00000008:00100000:2.0:1788572518.178442:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617c380 - object 0x280000401:552 offset 0<>12288 00000008:00010000:2.0:1788572518.199686:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc58ca8480 00000020:00200000:2.0:1788572518.199698:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:2.0:1788572518.199701:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:2.0:1788572518.199703:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1788572518.199709:0:2396:0:(cl_io.c:1302:cl_sub_dio_end()) finishing unaligned dio read aio->cda_bytes 9000 00000020:00200000:2.0:1788572518.199718:0:2396:0:(cl_io.c:1693:__ll_dio_user_copy()) count 9000, offset 0, pos 0, cdp_page_count 3 00000020:00200000:2.0:1788572518.199743:0:2396:0:(cl_io.c:1693:__ll_dio_user_copy()) count 4904, offset 0, pos 4096, cdp_page_count 3 00000020:00200000:2.0:1788572518.199747:0:2396:0:(cl_io.c:1693:__ll_dio_user_copy()) count 808, offset 0, pos 8192, cdp_page_count 3 00000020:00200000:2.0:1788572518.199759:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572518.199781:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc58ca8480 00000100:00100000:2.0:1788572518.199816:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2396:1875449863871232:192.168.202.132@tcp:3:lfs.0 00000080:00200000:1.0:1788572518.200284:0:110031:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:1.0:1788572518.200306:0:110031:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572518.200312:0:110031:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000020:00200000:1.0:1788572518.200316:0:110031:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572518.200330:0:110031:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200080:1.0:1788572518.200349:0:110031:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200080:1.0:1788572518.224063:0:110031:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=c00c66f3 arg=7ffebdb23464 00010000:00010000:1.0:1788572518.224092:0:110031:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70204000/0x58a604dcef352638 lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a221792715 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572518.224158:0:110031:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70204000/0x58a604dcef352638 lrc: 1/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221792715 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572518.224167:0:110031:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572518.224257:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863871360:192.168.202.132@tcp:103:lfs.0 00000100:00100000:1.0:1788572518.224393:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d750c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572518.231204:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863871360:192.168.202.132@tcp:103:lfs.0 00010000:00010000:2.0:1788572518.231233:0:110031:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70204000/0x58a604dcef352638 lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221792715 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572518.231312:0:110031:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67339c00 x1875449863870848/t4294983712(4294983712) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572518.231324:0:110031:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572518.231330:0:110031:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863871616/t0(0) o35->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572518.231350:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863871616:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572518.231423:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572518.239152:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863871616:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572518.239207:0:110031:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67339c00 x1875449863870848/t4294983712(4294983712) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572518.239272:0:110031:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 154000 00000002:00100000:2.0:1788572518.239342:0:110031:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6704a680 x1875449863870464/t4294983711(4294983711) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572518.239352:0:110031:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572518.239358:0:110031:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863871872/t0(0) o35->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572518.239381:0:110031:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863871872:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572518.239456:0:110031:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572518.242999:0:110031:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110031:1875449863871872:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572518.243040:0:110031:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6704a680 x1875449863870464/t4294983711(4294983711) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572518.285380:0:110036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:0.0:1788572518.285416:0:110036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.285437:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572518.285454:0:110036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572518.285463:0:110036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.285474:0:110036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:0.0:1788572518.285478:0:110036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:0.0:1788572518.285486:0:110036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e0383a63b58 00010000:00010000:0.0:1788572518.285490:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572518.285499:0:110036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572518.285516:0:110036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572518.285523:0:110036:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:0.0:1788572518.285529:0:110036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148), inode mode 41ed mask 1 00000080:00200000:0.0:1788572518.285533:0:110036:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f59b.sanity-sec, flags=9 00000080:00200008:0.0:1788572518.285540:0:110036:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572518.285546:0:110036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:0.0:1788572518.285590:0:110036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.285598:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572518.285607:0:110036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572518.285614:0:110036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.285618:0:110036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:0.0:1788572518.285620:0:110036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:0.0:1788572518.285624:0:110036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e0383a63cb8 00010000:00010000:0.0:1788572518.285627:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572518.285634:0:110036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788572518.285672:0:110036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:0.0:1788572518.285679:0:110036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.285685:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572518.285693:0:110036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572518.285699:0:110036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.285703:0:110036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:0.0:1788572518.285706:0:110036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:0.0:1788572518.285709:0:110036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e0383a63bf8 00010000:00010000:0.0:1788572518.285712:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572518.285747:0:110036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572518.285756:0:110036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572518.285759:0:110036:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:0.0:1788572518.285763:0:110036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148), inode mode 41ed mask 1 00000080:00200000:0.0:1788572518.285766:0:110036:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f59b.sanity-sec, flags=265 00000080:00200000:0.0:1788572518.285769:0:110036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), inode mode 81a4 mask 44 00000080:00200008:0.0:1788572518.285777:0:110036:0:(file.c:1090:ll_file_open()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 154400 00000002:00010000:0.0:1788572518.288536:0:110036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0104240114401 00010000:00010000:0.0:1788572518.288618:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110036 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.288629:0:110036:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110036 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572518.288642:0:110036:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572518.288656:0:110036:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67048a80 x1875449863872128/t0(0) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572518.288666:0:110036:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110036 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572518.288691:0:110036:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863872128:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572518.290618:0:110036:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fc00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572518.293466:0:110036:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863872128:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572518.293503:0:110036:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.293515:0:110036:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.293529:0:110036:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.293541:0:110036:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572518.293570:0:110036:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc67048a80 x1875449863872128/t4294983715(4294983715) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572518.293585:0:110036:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048a80 x1875449863872128/t4294983715(4294983715) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572518.293599:0:110036:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572518.293614:0:110036:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.293625:0:110036:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc67270000 type 13 mode 16 res 8589935645/2 (0 0) 00000080:00200000:0.0:1788572518.293632:0:110036:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572518.293708:0:110036:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00200000:0.0:1788572518.293757:0:110036:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 2 -> 2 00000080:00010000:0.0:1788572518.293763:0:110036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef352646 00000080:00010000:0.0:1788572518.293771:0:110036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352646 from it ffff9e0383a63bf0 00010000:00010000:0.0:1788572518.293775:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.293785:0:110036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788572518.293813:0:110036:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048a80 x1875449863872128/t4294983715(4294983715) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1788572518.293918:0:110036:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=c00c66f3 arg=7fff2b2d87f4 00000002:00010000:0.0:1788572518.293939:0:110036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: open flags 01004040000001 00010000:00010000:0.0:1788572518.293985:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 110036 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.293993:0:110036:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110036 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572518.294004:0:110036:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572518.294011:0:110036:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67048e00 x1875449863872384/t0(0) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572518.294020:0:110036:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x81000000000000 pid: 110036 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572518.294040:0:110036:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863872384:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572518.294182:0:110036:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723fc00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572518.302267:0:110036:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863872384:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572518.302309:0:110036:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302323:0:110036:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 4/1,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302335:0:110036:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302346:0:110036:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302360:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302368:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302375:0:110036:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 4/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572518.302388:0:110036:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff92fc67048e00 x1875449863872384/t4294983716(4294983716) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572518.302401:0:110036:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048e00 x1875449863872384/t4294983716(4294983716) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572518.302417:0:110036:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302436:0:110036:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc67270000 type 13 mode 2 res 8589935645/2 (0 0) 00000080:00200000:0.0:1788572518.302444:0:110036:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000002:00100000:0.0:1788572518.302463:0:110036:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048e00 x1875449863872384/t4294983716(4294983716) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572518.302473:0:110036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35264d 00000080:00010000:0.0:1788572518.302479:0:110036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35264d from it ffff9e0383a63cc0 00010000:00010000:0.0:1788572518.302483:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572518.302490:0:110036:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788572518.302769:0:110036:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), inode mode 81a4 mask 4 00000080:00200000:0.0:1788572518.302781:0:110036:0:(xattr.c:744:ll_xattr_get()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), xattr lov 00000080:00200008:0.0:1788572518.302826:0:110036:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572518.302835:0:110036:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:0.0:1788572518.302875:0:110036:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.302886:0:110036:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572518.302899:0:110036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788572518.302907:0:110036:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.302913:0:110036:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:0.0:1788572518.302917:0:110036:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:0.0:1788572518.302921:0:110036:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e0383a63cd8 00010000:00010000:0.0:1788572518.302925:0:110036:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572518.302933:0:110036:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 1 00000080:00200080:0.0:1788572518.303100:0:110036:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=66f4 arg=1 00000100:00100000:2.0:1788572518.367527:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:495b6acd-313a-4a79-a435-ab1ce65f91fe:2395:1875449863872512:192.168.202.132@tcp:3:lfs.0 00000008:00010000:2.0:1788572518.380144:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc45a76c00 00000020:00200000:2.0:1788572518.380156:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:2.0:1788572518.380160:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:2.0:1788572518.380162:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1788572518.380168:0:2395:0:(cl_io.c:1302:cl_sub_dio_end()) finishing unaligned dio read aio->cda_bytes 9000 00000020:00200000:2.0:1788572518.380176:0:2395:0:(cl_io.c:1693:__ll_dio_user_copy()) count 9000, offset 0, pos 0, cdp_page_count 3 00000020:00200000:2.0:1788572518.380182:0:2395:0:(cl_io.c:1693:__ll_dio_user_copy()) count 4904, offset 0, pos 4096, cdp_page_count 3 00000020:00200000:2.0:1788572518.380186:0:2395:0:(cl_io.c:1693:__ll_dio_user_copy()) count 808, offset 0, pos 8192, cdp_page_count 3 00000020:00200000:2.0:1788572518.380196:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572518.380219:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc45a76c00 00000100:00100000:2.0:1788572518.380244:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:495b6acd-313a-4a79-a435-ab1ce65f91fe:2395:1875449863872512:192.168.202.132@tcp:3:lfs.0 00000080:00200000:2.0:1788572518.391465:0:110036:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:2.0:1788572518.391483:0:110036:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572518.391488:0:110036:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000020:00200000:2.0:1788572518.391492:0:110036:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572518.391503:0:110036:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200080:2.0:1788572518.391522:0:110036:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200080:2.0:1788572518.404539:0:110036:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=c00c66f3 arg=7fff2b2d8804 00010000:00010000:2.0:1788572518.404576:0:110036:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.404705:0:110036:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 1/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572518.404714:0:110036:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572518.405460:0:110036:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863872768:192.168.202.132@tcp:103:lfs.0 00000100:00100000:2.0:1788572518.405666:0:110036:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572518.413818:0:110036:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863872768:192.168.202.132@tcp:103:lfs.0 00010000:00010000:2.0:1788572518.413853:0:110036:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3800/0x58a604dcef35264d lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 4 type: IBT gid 0 flags: 0x85809400000020 nid: local remote: 0xc9a239a221792754 expref: -99 pid: 110036 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572518.413927:0:110036:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67048e00 x1875449863872384/t4294983716(4294983716) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572518.413939:0:110036:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572518.413944:0:110036:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863873024/t0(0) o35->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572518.413966:0:110036:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863873024:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572518.414049:0:110036:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572518.422329:0:110036:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:495b6acd-313a-4a79-a435-ab1ce65f91fe:110036:1875449863873024:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572518.422381:0:110036:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67048e00 x1875449863872384/t4294983716(4294983716) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572518.422447:0:110036:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 154000 00010000:00010000:2.0:1788572518.422480:0:110036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572518.460771:0:110038:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:1.0:1788572518.460985:0:110038:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572518.461007:0:110038:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572518.461023:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788572518.461031:0:110038:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572518.461040:0:110038:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:1.0:1788572518.461045:0:110038:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:1.0:1788572518.461053:0:110038:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e03837cfbf8 00010000:00010000:1.0:1788572518.461057:0:110038:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572518.461390:0:110038:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572518.461414:0:110038:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572518.461423:0:110038:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:1.0:1788572518.461428:0:110038:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148), inode mode 41ed mask 1 00000080:00200000:1.0:1788572518.461432:0:110038:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f59b.sanity-sec, flags=265 00000080:00200000:1.0:1788572518.461436:0:110038:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), inode mode 81a4 mask 44 00000080:00200008:1.0:1788572518.461442:0:110038:0:(file.c:1090:ll_file_open()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 140000 00000080:00200000:1.0:1788572518.462712:0:110038:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), inode mode 81a4 mask 4 00000080:00200000:1.0:1788572518.462742:0:110038:0:(xattr.c:744:ll_xattr_get()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), xattr lov 00000080:00200008:1.0:1788572518.462785:0:110038:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572518.463325:0:110038:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:1.0:1788572518.463448:0:110038:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572518.463468:0:110038:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572518.463483:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572518.463492:0:110038:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572518.463498:0:110038:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:1.0:1788572518.463501:0:110038:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:1.0:1788572518.463508:0:110038:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfcd8 00010000:00010000:1.0:1788572518.463511:0:110038:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572518.463520:0:110038:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000080:00200080:2.0:1788572518.523713:0:110038:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=1 00000080:00200008:2.0:1788572518.523891:0:110038:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572518.523901:0:110038:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:2.0:1788572518.523957:0:110038:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572518.523981:0:110038:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572518.523998:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572518.524007:0:110038:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572518.524140:0:110038:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:2.0:1788572518.524145:0:110038:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:2.0:1788572518.524155:0:110038:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfcd8 00010000:00010000:2.0:1788572518.524158:0:110038:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572518.524167:0:110038:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 12 lru_type: 1 00000080:00200008:2.0:1788572518.524205:0:110038:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 0, count: 67108864 00000080:00200000:2.0:1788572518.524219:0:110038:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 0, bytes: 67108864 00000080:00200000:2.0:1788572518.524230:0:110038:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 1 00000020:00200000:2.0:1788572518.524238:0:110038:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d7480875[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:2.0:1788572518.524240:0:110038:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 67108864) 0 0 00000080:00200000:2.0:1788572518.524246:0:110038:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:2.0:1788572518.524260:0:110038:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 16383] 00020000:00200000:2.0:1788572518.524269:0:110038:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 67108864) 67108864, 67108864 00020000:00200000:2.0:1788572518.524273:0:110038:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572518.524305:0:110038:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 16383] 00000008:00010000:2.0:1788572518.524322:0:110038:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1788572518.524326:0:110038:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550630, osc lock ffff92fc58370ab0, flags 0x0 00000080:00200000:2.0:1788572518.524417:0:110038:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [0, 67108864) 00000001:00010000:2.0:1788572518.524430:0:110038:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:2.0:1788572518.524437:0:110038:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572518.524439:0:110038:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc705508b0, osc lock ffff92fc583717c0, flags 0x400001002 00010000:00010000:2.0:1788572518.524457:0:110038:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572518.524469:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572518.524480:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572518.524484:0:110038:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572518.524492:0:110038:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572518.524502:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572518.524509:0:110038:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:2.0:1788572518.524513:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbc000 (ffff92fc58255908/0) at cl_lock_release():245 00000020:00010000:2.0:1788572518.524516:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbc000 (ffff92fc58255908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572518.524519:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc705508b0 (ffff92fc6713bea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572518.524523:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbc000 (ffff92fc58255908/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572518.524526:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc705508b0 (ffff92fc58255908/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572518.524538:0:110038:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:2.0:1788572518.524670:0:110038:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), size=67108864 (max 536870912), offset=0=0x0, pages 16384 (max 131072), lockless, parallel 00000020:00200000:2.0:1788572518.524767:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:2.0:1788572518.524770:0:110038:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 9000 count 9000 file_offset 9000 00000020:00010000:2.0:1788572518.524779:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c038 (ffff92fc58255908/0) at cl_lock_release():245 00000020:00010000:2.0:1788572518.524781:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c038 (ffff92fc58255908/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572518.524783:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550630 (ffff92fc6713bea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572518.524910:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c038 (ffff92fc58255908/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572518.524915:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550630 (ffff92fc58255908/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572518.524921:0:110038:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000020:00200000:2.0:1788572518.524928:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1788572518.525016:0:107990:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc58ca8480 00000008:00100000:2.0:1788572518.525062:0:107990:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617c380 - object 0x2c0000401:537 offset 0<>12288 00000080:00200000:0.0:1788572518.531592:0:110038:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:0.0:1788572518.531611:0:110038:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572518.531617:0:110038:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000020:00200000:0.0:1788572518.531620:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572518.531631:0:110038:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200080:0.0:1788572518.531649:0:110038:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200080:0.0:1788572518.531773:0:110038:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=1 00000080:00200008:0.0:1788572518.531782:0:110038:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572518.531801:0:110038:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:0.0:1788572518.531844:0:110038:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572518.531864:0:110038:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572518.531880:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572518.531888:0:110038:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572518.531897:0:110038:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:0.0:1788572518.531900:0:110038:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:0.0:1788572518.531909:0:110038:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e03837cfcd8 00010000:00010000:0.0:1788572518.531913:0:110038:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572518.531923:0:110038:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 13 lru_type: 1 00000080:00200008:0.0:1788572518.531949:0:110038:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 9000, count: 67108864 00000080:00200000:0.0:1788572518.531957:0:110038:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 9000, bytes: 67108864 00000080:00200000:0.0:1788572518.531964:0:110038:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 1 00000020:00200000:0.0:1788572518.531972:0:110038:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d7480875[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:0.0:1788572518.531974:0:110038:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [9000, 67117864) 0 0 00000080:00200000:0.0:1788572518.531978:0:110038:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:0.0:1788572518.531991:0:110038:0:(file.c:2168:ll_file_io_generic()) Range lock [2, 16386] 00020000:00200000:0.0:1788572518.531998:0:110038:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 9000 chunk: [9000, 67117864) 67117864, 67108864 00020000:00200000:0.0:1788572518.532002:0:110038:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572518.532039:0:110038:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [2, 16386] 00000008:00010000:0.0:1788572518.532053:0:110038:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1788572518.532057:0:110038:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2db0, osc lock ffff92fc45932ab0, flags 0x0 00000080:00200000:0.0:1788572518.532064:0:110038:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [9000, 67117864) 00000001:00010000:0.0:1788572518.532248:0:110038:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:0.0:1788572518.532256:0:110038:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572518.532258:0:110038:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f23b0, osc lock ffff92fc45933c80, flags 0x400001002 00010000:00010000:0.0:1788572518.532274:0:110038:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572518.532286:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:0.0:1788572518.532307:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572518.532311:0:110038:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572518.532320:0:110038:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:0.0:1788572518.532431:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572518.532438:0:110038:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:0.0:1788572518.532442:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbc000 (ffff92fc58255908/0) at cl_lock_release():245 00000020:00010000:0.0:1788572518.532445:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbc000 (ffff92fc58255908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572518.532448:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f23b0 (ffff92fc6713bea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572518.532452:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbc000 (ffff92fc58255908/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572518.532455:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f23b0 (ffff92fc58255908/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572518.532466:0:110038:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:9000 00000080:00200000:0.0:1788572518.532471:0:110038:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), size=67108864 (max 536870912), offset=9000=0x2328, pages 16384 (max 131072), lockless, parallel, unaligned 00000020:00010000:0.0:1788572518.532483:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804c038 (ffff92fc58255908/0) at cl_lock_release():245 00000020:00010000:0.0:1788572518.532486:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804c038 (ffff92fc58255908/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572518.532488:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2db0 (ffff92fc6713bea8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572518.532491:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804c038 (ffff92fc58255908/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572518.532493:0:110038:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2db0 (ffff92fc58255908/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572518.532498:0:110038:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:0.0:1788572518.532503:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572518.532505:0:110038:0:(file.c:2205:ll_file_io_generic()) Range unlock [2, 16386] 00000080:00200000:0.0:1788572518.532510:0:110038:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572518.532513:0:110038:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:0.0:1788572518.532515:0:110038:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572518.532519:0:110038:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200080:0.0:1788572518.532525:0:110038:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200008:0.0:1788572518.562196:0:110038:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 140000 00010000:00010000:0.0:1788572518.562243:0:110038:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1788572518.680599:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2396:1875449863873280:192.168.202.132@tcp:101:lfs.0 00000080:00200000:1.0:1788572518.686266:0:110040:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 48 00000020:00010000:1.0:1788572518.686273:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc671746f0 (ffff92fc6713b7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572518.686277:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc671746f0 (ffff92fc6713b7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.686282:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049dcb0 (ffff92fc6713b488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.686286:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc671746f0 (ffff92fc6713b7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572518.686289:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049dcb0 (ffff92fc6713b7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572518.686309:0:110040:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:0 00000080:00200000:1.0:1788572518.686316:0:110040:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), size=67108864 (max 536870912), offset=0=0x0, pages 16384 (max 131072), lockless, parallel 00000020:00200000:1.0:1788572518.686374:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 3 items remaining. 00000080:00200000:1.0:1788572518.686377:0:110040:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 9000 count 9000 file_offset 9000 00000020:00010000:1.0:1788572518.686383:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc7002d218 (ffff92fc6713b7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572518.686386:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7002d218 (ffff92fc6713b7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.686388:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049d530 (ffff92fc6713b488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.686391:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7002d218 (ffff92fc6713b7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572518.686394:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049d530 (ffff92fc6713b7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572518.686399:0:110040:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000020:00200000:1.0:1788572518.686405:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1788572518.686421:0:103957:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc48227000 00000008:00100000:1.0:1788572518.686462:0:103957:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc6728a680 - object 0x280000401:552 offset 0<>12288 00000100:00100000:1.0:1788572518.686493:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572518.686506:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2394:1875449863873408:192.168.202.132@tcp:3:lfs.0 00000008:00010000:0.0:1788572518.690547:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc48227000 00000020:00200000:0.0:1788572518.690560:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 2 items remaining. 00000020:00200000:0.0:1788572518.690564:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000020:00200000:0.0:1788572518.690565:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1788572518.690573:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1788572518.690602:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc48227000 00000100:00100000:0.0:1788572518.690628:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:495b6acd-313a-4a79-a435-ab1ce65f91fe:2394:1875449863873408:192.168.202.132@tcp:3:lfs.0 00000080:00200000:1.0:1788572518.690889:0:110040:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 16383] 00000080:00200000:1.0:1788572518.690902:0:110040:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572518.690907:0:110040:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000020:00200000:1.0:1788572518.690910:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572518.690918:0:110040:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200080:1.0:1788572518.690937:0:110040:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200080:1.0:1788572518.691231:0:110040:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=2 00000080:00200008:1.0:1788572518.691242:0:110040:0:(file.c:6296:ll_getattr_dentry()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572518.691251:0:110040:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08),name=f59b.sanity-sec 00000002:00010000:1.0:1788572518.691300:0:110040:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572518.691320:0:110040:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788572518.692499:0:110040:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788572518.692512:0:110040:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572518.692523:0:110040:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) 00000080:00010000:1.0:1788572518.692527:0:110040:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665eec08) for lock 0x58a604dcef35262a 00000080:00010000:1.0:1788572518.692536:0:110040:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35262a from it ffff9e0383a5bcd8 00010000:00010000:1.0:1788572518.692540:0:110040:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788572518.692551:0:110040:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00000080:00200008:1.0:1788572518.693434:0:110040:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 9000, count: 67108864 00000080:00200000:1.0:1788572518.693449:0:110040:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 9000, bytes: 67108864 00000080:00200000:1.0:1788572518.693464:0:110040:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 2 00000020:00200000:1.0:1788572518.693473:0:110040:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d7480875[0x110, 2, [0x20000041d:0x2:0x0] hash] 00000020:00200000:1.0:1788572518.693476:0:110040:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [9000, 67117864) 0 0 00000080:00200000:1.0:1788572518.693481:0:110040:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 restore needed 0 00000080:00200000:1.0:1788572518.693496:0:110040:0:(file.c:2168:ll_file_io_generic()) Range lock [2, 16386] 00020000:00200000:1.0:1788572518.693505:0:110040:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 9000 chunk: [9000, 67117864) 67117864, 67108864 00020000:00200000:1.0:1788572518.693510:0:110040:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788572518.693527:0:110040:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [2, 16386] 00000008:00010000:1.0:1788572518.693543:0:110040:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1788572518.693548:0:110040:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4f30, osc lock ffff92fc47ff5430, flags 0x0 00000080:00200000:1.0:1788572518.693555:0:110040:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [9000, 67117864) 00000001:00010000:1.0:1788572518.693564:0:110040:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:1.0:1788572518.693569:0:110040:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572518.693571:0:110040:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4530, osc lock ffff92fc47ff51d0, flags 0x400001002 00010000:00010000:1.0:1788572518.694433:0:110040:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572518.694451:0:110040:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788572518.694465:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572518.694469:0:110040:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572518.694476:0:110040:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572518.694491:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572518.694500:0:110040:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 48 00000020:00010000:1.0:1788572518.694504:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc671746f0 (ffff92fc6713b7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572518.694508:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc671746f0 (ffff92fc6713b7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.694511:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4530 (ffff92fc6713b488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.694515:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc671746f0 (ffff92fc6713b7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572518.694518:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4530 (ffff92fc6713b7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572518.694528:0:110040:0:(vvp_io.c:893:vvp_io_read_start()) count:67108864 iocb pos:9000 00000080:00200000:1.0:1788572518.694534:0:110040:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08), size=67108864 (max 536870912), offset=9000=0x2328, pages 16384 (max 131072), lockless, parallel, unaligned 00000020:00010000:1.0:1788572518.694546:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc7002d218 (ffff92fc6713b7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1788572518.694549:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7002d218 (ffff92fc6713b7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.694552:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4f30 (ffff92fc6713b488/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572518.694554:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7002d218 (ffff92fc6713b7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572518.694557:0:110040:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4f30 (ffff92fc6713b7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572518.694562:0:110040:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000020:00200000:1.0:1788572518.694567:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572518.694570:0:110040:0:(file.c:2205:ll_file_io_generic()) Range unlock [2, 16386] 00000080:00200000:1.0:1788572518.695217:0:110040:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572518.695223:0:110040:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 0, restart: 0 00000020:00200000:1.0:1788572518.695227:0:110040:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572518.695232:0:110040:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200080:1.0:1788572518.695245:0:110040:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) cmd=400866b1 arg=0 00000080:00200008:1.0:1788572518.708536:0:110040:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665eec08), flags 140000 00010000:00010000:1.0:1788572518.708575:0:110040:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200008:0.0:1788572519.104280:0:110056:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572519.104305:0:110056:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:0.0:1788572519.104332:0:110056:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572519.104351:0:110056:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572519.104367:0:110056:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572519.104376:0:110056:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572519.104383:0:110056:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:0.0:1788572519.104387:0:110056:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:0.0:1788572519.104395:0:110056:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e0383ab3cb8 00010000:00010000:0.0:1788572519.104399:0:110056:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572519.104408:0:110056:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572519.300938:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572521.019952:0:110064:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572521.019965:0:110064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:1.0:1788572521.019989:0:110064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572521.020145:0:110064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572521.020161:0:110064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572521.020169:0:110064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572521.020178:0:110064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:1.0:1788572521.020182:0:110064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:1.0:1788572521.020189:0:110064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e0383adbcb8 00010000:00010000:1.0:1788572521.020193:0:110064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572521.020201:0:110064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200008:1.0:1788572521.021202:0:110064:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572521.021212:0:110064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988),name=/ 00000002:00010000:1.0:1788572521.021228:0:110064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572521.021241:0:110064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572521.021254:0:110064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572521.021262:0:110064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572521.021268:0:110064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00010000:1.0:1788572521.021271:0:110064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e1988) for lock 0x58a604dcef35261c 00000080:00010000:1.0:1788572521.021276:0:110064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35261c from it ffff9e0383adbcb8 00010000:00010000:1.0:1788572521.021280:0:110064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572521.021301:0:110064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:0.0:1788572521.060357:0:71771:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00020000:00200000:0.0:1788572521.060371:0:71771:0:(lov_io.c:655:lov_io_mirror_init()) [0x20000041d:0x2:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:0.0:1788572521.060375:0:71771:0:(lov_io.c:709:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:0.0:1788572521.060381:0:71771:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:0.0:1788572521.060410:0:71771:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572521.060517:0:110064:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e1988) 00000080:00200000:1.0:1788572521.060549:0:110064:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e1148) 00000080:00200000:1.0:1788572521.060580:0:110064:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00020000:00200000:1.0:1788572521.060586:0:110064:0:(lov_io.c:655:lov_io_mirror_init()) [0x20000041d:0x2:0x0]: flr state: 1, move mirror from 1 to 1, have retried: 0, mirror count: 2 00020000:00200000:1.0:1788572521.060588:0:110064:0:(lov_io.c:709:lov_io_mirror_init()) use delayed RPC state for this IO 00020000:00200000:1.0:1788572521.060591:0:110064:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x10 00000080:00200000:1.0:1788572521.060602:0:110064:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572521.060606:0:110064:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665eec08) 00000002:00100000:1.0:1788572521.060670:0:110064:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67048a80 x1875449863872128/t4294983715(4294983715) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572521.060684:0:110064:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc58389800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572521.060691:0:110064:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6728b480 x1875449863873664/t0(0) o35->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572521.060716:0:110064:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863873664:192.168.202.132@tcp:35:umount.0 00000100:00100000:1.0:1788572521.060896:0:110064:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d753c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.069367:0:110064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863873664:192.168.202.132@tcp:35:umount.0 00000002:00100000:1.0:1788572521.069410:0:110064:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67048a80 x1875449863872128/t4294983715(4294983715) o101->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572534 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1788572521.069518:0:110064:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 9000 blocks 24 [cma]time: 1788572511 1788572501 1788572496 by oinfo size 9000 blocks 24 [cma]time 1788572511 1788572501 1788572518 ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 1/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:1.0:1788572521.069568:0:110064:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 9000 blocks 24 [cma]time: 1788572511 1788572511 1788572496 by oinfo size 9000 blocks 24 [cma]time 1788572511 1788572511 1788572518 ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 1/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572521.069621:0:110064:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc58389800 (sb ffff92fc58389800) 00010000:00010000:1.0:1788572521.069661:0:110064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069670:0:110064:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069677:0:110064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069684:0:110064:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069811:0:110064:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572521.069818:0:110064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069825:0:110064:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069831:0:110064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.069836:0:110064:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070279:0:110000:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788572521.070296:0:110000:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070303:0:110000:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367600 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572521.070307:0:110000:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070315:0:110000:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572521.070322:0:110000:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572521.070328:0:110000:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572521.070330:0:110000:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:2.0:1788572521.070366:0:110000:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070373:0:110000:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572521.070551:0:110000:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070564:0:110000:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3525b3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792682 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070579:0:110000:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572521.070583:0:110000:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070588:0:110000:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367800 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572521.070592:0:110000:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070599:0:110000:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 4/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572521.070605:0:110000:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 4/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572521.070610:0:110000:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572521.070612:0:110000:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 00010000:00010000:2.0:1788572521.070625:0:110000:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.070630:0:110000:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572521.070661:0:110064:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 10000000:01000000:1.0:1788572521.070666:0:110064:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572521.070696:0:109999:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572521.070870:0:110064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef3525eb lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792690 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.070879:0:110064:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef3525eb lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792690 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.070886:0:110064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef3525eb lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792690 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.070893:0:110064:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef3525eb lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792690 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.070909:0:110064:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:01000000:0.0:1788572521.070913:0:110064:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:0.0:1788572521.070917:0:110064:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:0.0:1788572521.071481:0:110064:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc67146800 active: 1 00020000:01000000:0.0:1788572521.071486:0:110064:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc67145800 active: 1 00020000:01000000:0.0:1788572521.071489:0:110064:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:0.0:1788572521.071495:0:110064:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc58389800 00020000:01000000:0.0:1788572521.071497:0:110064:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc58389800: disconnecting target lustre-OST0001-osc-ffff92fc58389800 00000100:00080000:0.0:1788572521.071509:0:110064:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-OST0001_UUID 00010000:00010000:0.0:1788572521.071542:0:110064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572521.071577:0:110064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572521.071585:0:110064:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572521.071596:0:110064:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f0c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572521.071604:0:110064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863874048:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572521.071613:0:110064:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572521.071619:0:110064:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572521.071623:0:110064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff92fc58389800 lock: 00000000f37817dd/0x58a604dcef35263f lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x219:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792723 expref: -99 pid: 110031 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00080000:0.0:1788572521.071709:0:110064:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1788572521.071795:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863874176:192.168.202.132@tcp:9:umount.0 00010000:00010000:1.0:1788572521.072491:0:109999:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572521.073513:0:110065:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572521.073518:0:110065:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572521.073520:0:110065:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572521.079312:0:110000:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.079331:0:110000:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3525e4 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792689 expref: -99 pid: 109988 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.079359:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572521.079361:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572521.079363:0:110000:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572521.084997:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572521.085099:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863874176:192.168.202.132@tcp:9:umount.0 00000100:00080000:0.0:1788572521.085185:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572521.085190:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc58389800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572521.085195:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572521.085405:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352615 00000020:00000080:0.0:1788572521.085423:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc67145800/LOV_OSC_UUID 00000020:01000000:0.0:1788572521.085513:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc58389800 (flags='') 00000020:00000080:0.0:1788572521.085518:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572521.085527:0:110064:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc58389800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572521.085532:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc48cc99c0) has no exports 00000020:00000080:2.0:1788572521.086140:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67145800/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc58389800 00000008:01000000:0.0:1788572521.086221:0:110064:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc58389800: client import never connected 00000100:00080000:0.0:1788572521.086225:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572521.086227:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc58389800: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572521.086232:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572521.086443:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572521.086450:0:110064:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00020000:01000000:0.0:1788572521.086454:0:110064:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc58389800 00020000:01000000:0.0:1788572521.086456:0:110064:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc58389800: disconnecting target lustre-OST0000-osc-ffff92fc58389800 00000100:00080000:0.0:1788572521.086466:0:110064:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-OST0000_UUID 00010000:00010000:0.0:1788572521.086497:0:110064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572521.086547:0:110064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572521.086554:0:110064:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572521.086568:0:110064:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f0c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572521.086576:0:110064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863874432:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572521.086586:0:110064:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572521.086592:0:110064:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572521.086595:0:110064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc58389800 lock: 00000000854823eb/0x58a604dcef352654 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792777 expref: -99 pid: 110040 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00080000:0.0:1788572521.086677:0:110064:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:2.0:1788572521.087190:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990a800 for lustre-OST0001-osc-ffff92fc58389800 00000020:00000080:2.0:1788572521.087199:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc4990c800/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:2.0:1788572521.087202:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990c800/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-OST0001-osc-ffff92fc58389800 00000020:01000000:2.0:1788572521.087209:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc58389800 (495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000100:00080000:0.0:1788572521.089341:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572521.089345:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc58389800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572521.089351:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572521.089482:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35260e 00000020:00000080:0.0:1788572521.089499:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc67146800/LOV_OSC_UUID 00000020:01000000:0.0:1788572521.089581:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc58389800 (flags='') 00000020:00000080:0.0:1788572521.089586:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572521.089595:0:110064:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc58389800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572521.089601:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc48ccb380) has no exports 00000020:00000080:2.0:1788572521.090125:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67146800/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc58389800 00000008:01000000:0.0:1788572521.090256:0:110064:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc58389800: client import never connected 00000100:00080000:0.0:1788572521.090260:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572521.090263:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc58389800: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572521.090267:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572521.090484:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572521.090491:0:110064:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:00000080:0.0:1788572521.090495:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352607 00000020:00000080:0.0:1788572521.090501:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc67141000/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:1.0:1788572521.090526:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990a000 for lustre-OST0000-osc-ffff92fc58389800 00000020:00000080:1.0:1788572521.090533:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc4990b000/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:1.0:1788572521.090535:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990b000/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-OST0000-osc-ffff92fc58389800 00000020:01000000:1.0:1788572521.090543:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc58389800 (495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000100:00080000:0.0:1788572521.090985:0:110064:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-MDT0000_UUID 00000020:00000080:1.0:1788572521.091086:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67141000/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-clilov-ffff92fc58389800 00010000:00010000:0.0:1788572521.091364:0:110064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572521.091418:0:110064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572521.091425:0:110064:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572521.091440:0:110064:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f0c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572521.091449:0:110064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863874688:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572521.091459:0:110064:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572521.091465:0:110064:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572521.091469:0:110064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc702f4200/0x58a604dcef35261c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217926ba expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572521.091919:0:110064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572521.091944:0:110064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572521.091957:0:110064:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572521.091964:0:110064:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f0c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572521.091969:0:110064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863874816:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572521.091976:0:110064:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572521.091980:0:110064:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572521.091983:0:110064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc70367e00/0x58a604dcef352623 lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217926d6 expref: -99 pid: 110031 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572521.092155:0:110064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572521.092167:0:110064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788572521.092184:0:110064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572521.092191:0:110064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788572521.092196:0:110064:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:0.0:1788572521.092202:0:110064:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6723f0c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572521.092207:0:110064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job umount:495b6acd-313a-4a79-a435-ab1ce65f91fe:110064:1875449863874944:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572521.092212:0:110064:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572521.092216:0:110064:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572521.092219:0:110064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3400/0x58a604dcef352646 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a22179273f expref: -99 pid: 110036 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572521.092235:0:110064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc58389800 lock: ffff92fc51bc3200/0x58a604dcef35262a lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217926f2 expref: -99 pid: 110031 timeout: 0 lvb_type: 3 lru_score: 16 lru_type: 1 00000100:00080000:0.0:1788572521.093276:0:110064:0:(import.c:65:import_set_state_nolock()) ffff92fc5106b000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572521.095973:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5106b000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572521.096128:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863875072:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572521.096166:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572521.096172:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc58389800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572521.096180:0:110064:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67049500 x1875449863875072/t4294983719(4294983719) o39->lustre-MDT0000-mdc-ffff92fc58389800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572531 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572521.096203:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572521.098697:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3525f9 00000020:00000080:0.0:1788572521.098715:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc481c2800/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:1.0:1788572521.098981:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc481c2800/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-MDT0000-mdc-ffff92fc58389800 00000100:00080000:0.0:1788572521.099962:0:110064:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 495b6acd-313a-4a79-a435-ab1ce65f91fe->lustre-MDT0001_UUID 00000100:00080000:0.0:1788572521.101611:0:110064:0:(import.c:65:import_set_state_nolock()) ffff92fc6714a800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572521.101682:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863875200:192.168.202.132@tcp:39:umount.0 00000100:00100000:1.0:1788572521.115656:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc58389800: committing for last_committed 4294976748 gen 1 00000100:00080000:1.0:1788572521.115673:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc6714a800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572521.115701:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:495b6acd-313a-4a79-a435-ab1ce65f91fe:2392:1875449863875200:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572521.115836:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572521.115841:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc58389800: committing for last_committed 4294976748 gen 2 00000100:00100000:0.0:1788572521.115846:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572521.120697:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352600 00000020:00000080:0.0:1788572521.120714:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc5824f000/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:1.0:1788572521.123508:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824f000/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-MDT0001-mdc-ffff92fc58389800 00000020:00000080:1.0:1788572521.126858:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3525f2 00000020:00000080:1.0:1788572521.126875:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc481c6800/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:01000000:1.0:1788572521.126904:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc58389800 (flags='') 00000020:00000080:1.0:1788572521.126915:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572521.126922:0:110064:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc58389800: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572521.126926:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc51cd99c0) has no exports 00000020:00000080:1.0:1788572521.126989:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572521.127005:0:110064:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:00000080:1.0:1788572521.127008:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc58c8a000/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:1.0:1788572521.127012:0:110064:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58c8a000/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-clilov-ffff92fc58389800 00000020:01000000:1.0:1788572521.127018:0:110064:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc58389800 (495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:1.0:1788572521.127180:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc58389800 (flags='') 00000020:00000080:1.0:1788572521.127183:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572521.127186:0:110064:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc58389800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572521.127189:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc51cdb380) has no exports 00000020:00000080:1.0:1788572521.127228:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572521.127231:0:110064:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:1.0:1788572521.127234:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc58389800 (flags='') 00000020:00000080:1.0:1788572521.127237:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572521.127239:0:110064:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc58389800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572521.127241:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc51cdcd40) has no exports 00000020:00000080:0.0:1788572521.127822:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc481c6800/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-clilmv-ffff92fc58389800 00000020:00000080:0.0:1788572521.127828:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc58c8d800/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:0.0:1788572521.127831:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58c8d800/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-clilmv-ffff92fc58389800 00000020:01000000:0.0:1788572521.127841:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc58389800 (495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000008:01000000:1.0:1788572521.128067:0:110064:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc58389800: client import never connected 00000100:00080000:1.0:1788572521.128072:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572521.128076:0:110064:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc58389800: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572521.128083:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572521.133245:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572521.133258:0:110064:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc58389800 (uuid 495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:01000000:1.0:1788572521.133263:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc58389800 (flags='') 00000020:00000080:1.0:1788572521.133267:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572521.133271:0:110064:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc58389800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572521.133276:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc48cccd40) has no exports 00000020:00000080:2.0:1788572521.136346:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5106b000 for lustre-MDT0000-mdc-ffff92fc58389800 00000020:00000080:2.0:1788572521.136362:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc5106f000/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:2.0:1788572521.136365:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5106f000/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-MDT0000-mdc-ffff92fc58389800 00000020:01000000:2.0:1788572521.136379:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc58389800 (495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000020:00000080:0.0:1788572521.161305:0:71771:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc6714a800 for lustre-MDT0001-mdc-ffff92fc58389800 00000020:00000080:0.0:1788572521.161322:0:71771:0:(genops.c:885:class_export_put()) final put ffff92fc6714f000/495b6acd-313a-4a79-a435-ab1ce65f91fe 00000020:00000080:0.0:1788572521.161325:0:71771:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6714f000/495b6acd-313a-4a79-a435-ab1ce65f91fe for lustre-MDT0001-mdc-ffff92fc58389800 00000020:01000000:0.0:1788572521.161353:0:71771:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc58389800 (495b6acd-313a-4a79-a435-ab1ce65f91fe) 00000100:00100000:1.0:1788572521.163528:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:9d8e3785-0a8f-445a-94cc-8fe7d50612c2:2392:1875449863875456:192.168.202.132@tcp:251:umount.0 00000100:00080000:1.0:1788572521.167186:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572521.167191:0:110064:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572521.167193:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572521.167195:0:110064:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572521.167206:0:110064:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3525a5 00000020:00000080:1.0:1788572521.167216:0:110064:0:(genops.c:885:class_export_put()) final put ffff92fc51c43800/9d8e3785-0a8f-445a-94cc-8fe7d50612c2 00000020:01000000:1.0:1788572521.167238:0:110064:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:1.0:1788572521.167245:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572521.167253:0:110064:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572521.167257:0:110064:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc51cd8000) has no exports 10000000:01000000:2.0:1788572521.167677:0:110002:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:1.0:1788572521.168096:0:110064:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:1.0:1788572521.168099:0:110064:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572521.168102:0:110064:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572521.168104:0:110064:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572521.168106:0:110064:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572521.168159:0:110064:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572521.168166:0:110064:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 9d8e3785-0a8f-445a-94cc-8fe7d50612c2) 00000020:01000004:1.0:1788572521.168172:0:110064:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc58389800 1 00000020:01000004:1.0:1788572521.168176:0:110064:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc58388800 00010000:00010000:1.0:1788572521.181114:0:110065:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572521.182124:0:109999:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:1.0:1788572521.182498:0:110000:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00000100:00100000:1.0:1788572521.182786:0:109989:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc49296000:109989] 0 exiting: rc = 1 00000100:00100000:1.0:1788572521.182840:0:109990:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc49ba8200:109990] 1 exiting: rc = 0 00000100:00100000:1.0:1788572521.183551:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.183560:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.198155:0:109995:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc610c5600:109995] 2 exiting: rc = 0 00000100:00100000:1.0:1788572521.198262:0:109994:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc610c4400:109994] 1 exiting: rc = 0 00000100:00100000:1.0:1788572521.198334:0:109993:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc49ba8600:109993] 0 exiting: rc = 1 00000100:00100000:2.0:1788572521.204604:0:109996:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc610c4800:109996] 0 exiting: rc = 1 00000100:00100000:1.0:1788572521.211697:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211707:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211712:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211718:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211960:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211966:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211972:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211977:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211982:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211988:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.211993:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212000:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212004:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212010:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212017:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212021:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212027:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212033:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212038:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212047:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212052:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212057:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212062:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212067:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212073:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212079:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212084:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212090:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212095:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212101:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212106:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212112:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212117:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212122:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212127:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212132:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212136:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212141:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212147:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212152:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212157:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212162:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212167:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212173:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212177:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212182:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212189:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212194:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212199:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212204:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212209:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212214:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212219:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212224:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212229:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212235:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212241:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212246:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212252:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212257:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212262:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212267:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212273:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212277:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212374:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212381:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212386:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212391:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212396:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212401:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212406:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212411:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212417:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212432:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212437:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212444:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212449:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212454:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212458:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212464:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212469:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212475:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212480:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212485:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212490:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212494:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212499:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212504:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212509:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212514:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212518:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212523:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212528:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212533:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212537:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212542:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212547:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212552:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212557:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212562:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212567:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212572:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212577:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212583:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212588:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212593:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212597:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212602:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212607:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212612:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212616:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212621:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212626:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212630:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212635:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212639:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212645:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212649:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212654:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212660:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212671:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212677:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212682:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212688:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212693:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212698:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212703:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572521.212708:0:71771:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:0.0:1788572521.453247:0:110078:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:0.0:1788572521.453252:0:110078:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:0.0:1788572521.453253:0:110078:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:0.0:1788572521.453255:0:110078:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:0.0:1788572521.454206:0:110078:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc5824a800 00000080:01000004:0.0:1788572521.454219:0:110078:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:0.0:1788572521.454235:0:110078:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:0.0:1788572521.454264:0:110078:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:0.0:1788572521.454277:0:110078:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc70034d40) 00000020:00000080:0.0:1788572521.455501:0:110078:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:0.0:1788572521.455704:0:110078:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572521.455753:0:110078:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572521.456120:0:110078:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:0.0:1788572521.456127:0:110078:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572521.458229:0:110078:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572521.458244:0:110078:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572521.458867:0:110078:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:2.0:1788572521.458875:0:110078:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:0.0:1788572521.459356:0:110078:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572521.459363:0:110078:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:0.0:1788572521.460229:0:110078:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572521.460619:0:110078:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572521.461630:0:110078:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:0.0:1788572521.461639:0:110078:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:0.0:1788572521.476681:0:110078:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:0.0:1788572521.476689:0:110078:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:1.0:1788572521.481553:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572521.481560:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572521.481563:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788572521.482692:0:110078:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc67141800@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572521.483118:0:110078:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:1.0:1788572521.483126:0:110078:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 00010000:00010000:1.0:1788572521.483993:0:110089:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572521.485542:0:110092:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:1.0:1788572521.485648:0:110078:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid 124343f7-953d-45b3-8c00-3a0ab68d71b0) 00000020:00000080:1.0:1788572521.485677:0:110078:0:(genops.c:1385:class_connect()) connect: client 124343f7-953d-45b3-8c00-3a0ab68d71b0, cookie 0x58a604dcef352662 00000100:00080000:1.0:1788572521.485694:0:110078:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572521.485706:0:110078:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc67141800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572521.486136:0:110078:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449863875584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572521.486160:0:110078:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 124343f7-953d-45b3-8c00-3a0ab68d71b0->MGS 00000080:00200000:1.0:1788572521.486181:0:110078:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc5824a800 (sb ffff92fc5824a800) 00000080:01000000:1.0:1788572521.486240:0:110078:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:1.0:1788572521.486242:0:110078:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:1.0:1788572521.486246:0:110078:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:1.0:1788572521.486254:0:110078:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: b4a32cca-995b-4bf1-a799-13b256fbfebe 00000100:00100000:0.0:1788572521.486814:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:124343f7-953d-45b3-8c00-3a0ab68d71b0:2392:1875449863875584:192.168.202.132@tcp:250:mount.lustre.0 10000000:01000000:1.0:1788572521.488592:0:110078:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:1.0:1788572521.488598:0:110078:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc5824a800 10000000:01000000:1.0:1788572521.488604:0:110078:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc70034d40 10000000:01000000:1.0:1788572521.488609:0:110078:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788572521.488619:0:110078:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc70034d40 from 1 10000000:01000000:1.0:1788572521.488622:0:110078:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788572521.488669:0:110078:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.488682:0:110078:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.488694:0:110078:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572521.491338:0:110078:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67289500 x1875449863875712/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572521.491360:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d750c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572521.499979:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572521.499995:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:1.0:1788572521.500003:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:1.0:1788572521.500008:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:1.0:1788572521.500013:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc67141800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:1.0:1788572521.500016:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:1.0:1788572521.500031:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:1.0:1788572521.500042:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000004fd2583c): req@ffff92fc67289500 x1875449863875712/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572521.500058:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:124343f7-953d-45b3-8c00-3a0ab68d71b0:2392:1875449863875584:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:1.0:1788572521.500166:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d750c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.510019:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863875712:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572521.510053:0:110078:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.510068:0:110078:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572521.510175:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863875840:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572521.510260:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d750c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.512954:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863875840:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:1.0:1788572521.513008:0:110078:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788572521.513025:0:110078:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.513036:0:110078:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572521.513045:0:110078:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc5824a800 10000000:01000000:1.0:1788572521.513050:0:110078:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:1.0:1788572521.513055:0:110078:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc5824a800 10000000:01000000:1.0:1788572521.513058:0:110078:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788572521.513061:0:110078:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc5824a800 10000000:01000000:1.0:1788572521.513063:0:110078:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:1.0:1788572521.513067:0:110078:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc5824a800 from 1 10000000:01000000:1.0:1788572521.513070:0:110078:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572521.513103:0:110078:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.513112:0:110078:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.513122:0:110078:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572521.513143:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863875968:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788572521.513220:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.528952:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863875968:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572521.528994:0:110078:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572521.529009:0:110078:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572521.529104:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876096:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572521.530608:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.533658:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876096:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572521.533750:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876224:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572521.533964:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.542017:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876224:192.168.202.132@tcp:503:mount.lustre.0 00000020:01000000:0.0:1788572521.607214:0:110078:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:0.0:1788572521.607239:0:110078:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572521.607261:0:110078:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.607273:0:110078:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572521.607298:0:110078:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc5824a800 from 1 10000000:01000000:0.0:1788572521.607301:0:110078:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572521.607345:0:110078:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.607354:0:110078:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572521.607364:0:110078:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572521.611002:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876736:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572521.611122:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3b0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572521.616520:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876736:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572521.616552:0:110078:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.616643:0:110078:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572521.619251:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876864:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572521.619549:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c12000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572521.630525:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876864:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572521.631364:0:110078:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572521.631406:0:110078:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.631417:0:110078:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572521.631425:0:110078:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc5824a800 from 1 10000000:01000000:2.0:1788572521.631428:0:110078:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572521.631470:0:110078:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.631478:0:110078:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.631487:0:110078:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572521.631510:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876992:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572521.631590:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c12000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572521.634559:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863876992:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572521.634596:0:110078:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217927a1 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572521.634610:0:110078:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217927a1 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572521.634713:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863877120:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572521.634825:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c12000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572521.641984:0:110078:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863877120:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572521.642052:0:110078:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:124343f7-953d-45b3-8c00-3a0ab68d71b0:110078:1875449863877248:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572521.642151:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c12000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572521.645964:0:110101:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:124343f7-953d-45b3-8c00-3a0ab68d71b0:110101:1875449863877376:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572521.646609:0:110101:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5a80 going to sleep for 11 seconds 00000020:01000000:0.0:1788572521.652018:0:110101:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572521.652032:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572521.652034:0:110101:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:0.0:1788572521.652042:0:110101:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc5824a800 00000020:00000080:0.0:1788572521.652045:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:0.0:1788572521.652052:0:110101:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572521.652055:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572521.652056:0:110101:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:0.0:1788572521.652059:0:110101:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572521.652061:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572521.652062:0:110101:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:0.0:1788572521.652065:0:110101:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc5824a800 00000020:00000080:0.0:1788572521.652067:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:0.0:1788572521.652140:0:110101:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572521.652144:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572521.652146:0:110101:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:0.0:1788572521.652148:0:110101:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572521.652151:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572521.652153:0:110101:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:0.0:1788572521.652156:0:110101:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc5824a800 00000020:00000080:0.0:1788572521.652157:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:2.0:1788572521.661512:0:110101:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 9346us 00000100:00100000:2.0:1788572521.661575:0:110101:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:124343f7-953d-45b3-8c00-3a0ab68d71b0:110101:1875449863877504:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572521.661675:0:110101:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42c12600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572521.665036:0:110101:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job llog_process_th:124343f7-953d-45b3-8c00-3a0ab68d71b0:110101:1875449863877504:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572521.665082:0:110101:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572521.665091:0:110101:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572521.665093:0:110101:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:2.0:1788572521.665098:0:110101:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000100:00100000:2.0:1788572521.666239:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863877632:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572521.666350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863877888:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572521.676309:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5824a800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572521.676329:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc5824a800: connect to target with instance 1 00000100:00080000:2.0:1788572521.676340:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc5824a800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572521.676353:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1788572521.676357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58c8a000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788572521.676781:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000000887744f): req@ffff92fc6733a680 x1875449863878144/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572521.676801:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000001f6065c3): req@ffff92fc6733b480 x1875449863877760/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572521.676815:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863877632:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572521.676831:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5824a800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572521.676837:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc5824a800: connect to target with instance 1 00000100:00080000:2.0:1788572521.676842:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc5824a800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572521.676848:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:2.0:1788572521.676850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc492e1800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:2.0:1788572521.676944:0:110078:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384eb40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572521.677225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863877888:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572521.688715:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b4a32cca-995b-4bf1-a799-13b256fbfebe:0:1875449863878016:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572521.691807:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b4a32cca-995b-4bf1-a799-13b256fbfebe:0:1875449863877760:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572521.692402:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863878400:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572521.692804:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863878528:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572521.700576:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc5824a800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572521.700587:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc5824a800: connect to target with instance 1 00000100:00080000:2.0:1788572521.700598:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc5824a800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572521.700610:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572521.700613:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2a800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572521.700619:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1788572521.700624:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572521.700629:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572521.700640:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863878400:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572521.700663:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc5824a800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572521.700668:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc5824a800: connect to target with instance 1 00000100:00080000:2.0:1788572521.700673:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc5824a800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572521.700677:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788572521.700679:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572521.700682:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:2.0:1788572521.700684:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572521.700686:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572521.700692:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863878528:192.168.202.132@tcp:8:mount.lustre.0 00010000:00010000:1.0:1788572521.953329:0:110132:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:1.0:1788572521.953372:0:110132:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc5824a800 00010000:00010000:1.0:1788572521.953401:0:110132:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc5824a800 00010000:00010000:1.0:1788572521.953430:0:110132:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc5824a800 00010000:00010000:1.0:1788572521.953458:0:110132:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc5824a800 00010000:00010000:2.0:1788572522.502009:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572522.708631:0:110139:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572522.708673:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.708909:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.708922:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.708933:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.708967:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863878912:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.709108:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.712460:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863878912:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.712485:0:110139:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712496:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712507:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712520:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712531:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712538:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712545:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.712557:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617e680 x1875449863878912/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.712576:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712590:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572522.712597:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572522.712626:0:110139:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572522.712630:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:2.0:1788572522.712634:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572522.712638:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383b5bbf8 00010000:00010000:2.0:1788572522.712641:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.712649:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.712673:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.712691:0:110139:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665d9988), flags=25 00000080:00200000:2.0:1788572522.712696:0:110139:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665d9988), intent=0 00000002:00010000:2.0:1788572522.712716:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572522.713541:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.713553:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.713563:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.713590:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879040:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.713711:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.721115:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879040:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.721143:0:110139:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.721168:0:110139:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.722911:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.722925:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.722937:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.722955:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.722963:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.722971:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.722992:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617e680 x1875449863879040/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.723004:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.723138:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:2.0:1788572522.723146:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572522.723249:0:110139:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908) 00000080:00200000:2.0:1788572522.723257:0:110139:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572522.723286:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef3526e0 00000080:00010000:2.0:1788572522.723299:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e0908) 00000080:00010000:2.0:1788572522.723303:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526e0 from it ffff9e0383b5bac8 00010000:00010000:2.0:1788572522.723306:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.723316:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366800/0x58a604dcef3526e0 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927e7 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.723352:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.723371:0:110139:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), file ffff92fc49154300, open_flags d900, mode 0 opened 0 00000080:00200000:2.0:1788572522.723378:0:110139:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), intent=open 00000100:00100000:2.0:1788572522.723466:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879168:192.168.202.132@tcp:900:lfs.0 00000100:00100000:2.0:1788572522.723611:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.734850:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879168:192.168.202.132@tcp:900:lfs.0 00000002:00010000:2.0:1788572522.734900:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0154401 00010000:00010000:2.0:1788572522.734965:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.734975:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572522.734986:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572522.734995:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e680 x1875449863879424/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572522.735004:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.735097:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879424:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.735398:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.742782:0:110139:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5824a800: committing for last_committed 4294983720 gen 1 00000100:00100000:2.0:1788572522.742802:0:110139:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617e680 x1875449863879424/t4294983720(4294983720) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/880 e 0 to 0 dl 1788572538 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572522.742814:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879424:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.742839:0:110139:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742860:0:110139:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742868:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742886:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742897:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742911:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742920:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742929:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.742941:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617e680 x1875449863879424/t4294983720(4294983720) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/880 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572522.742955:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e680 x1875449863879424/t4294983720(4294983720) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/880 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.742969:0:110139:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572522.742984:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.742996:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 4 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572522.743003:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572522.743038:0:110139:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) 00000080:00200000:2.0:1788572522.743132:0:110139:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00200000:2.0:1788572522.743156:0:110139:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:2.0:1788572522.743181:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3526e7 00000080:00010000:2.0:1788572522.743220:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) 00000080:00200008:2.0:1788572522.743237:0:110139:0:(file.c:1090:ll_file_open()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), flags 154400 00000002:00100000:2.0:1788572522.743267:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e680 x1875449863879424/t4294983720(4294983720) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/880 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572522.743320:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526e7 from it ffff92fc51c92c00 00010000:00010000:2.0:1788572522.743324:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.743333:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.743350:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), inode mode 81a4 mask 44 00000080:00200000:2.0:1788572522.743455:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), inode mode 81a4 mask 4 00000080:00200000:2.0:1788572522.743462:0:110139:0:(xattr.c:744:ll_xattr_get()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), xattr lov 00000080:00200008:2.0:1788572522.743491:0:110139:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572522.743498:0:110139:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572522.743535:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.743544:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.743556:0:110139:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.743570:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572522.743575:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) 00000080:00010000:2.0:1788572522.743579:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3526e7 00000080:00010000:2.0:1788572522.743583:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526e7 from it ffff9e0383b5bcd8 00010000:00010000:2.0:1788572522.743587:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.743595:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000080:00200008:2.0:1788572522.743639:0:110139:0:(file.c:479:ll_file_release()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), flags 154000 00000002:00100000:2.0:1788572522.743682:0:110139:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617e680 x1875449863879424/t4294983720(4294983720) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/880 e 0 to 0 dl 1788572538 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572522.743694:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572522.743699:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ce00 x1875449863879680/t0(0) o35->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572522.743753:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879680:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572522.743836:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.747154:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879680:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572522.747202:0:110139:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617e680 x1875449863879424/t4294983720(4294983720) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/880 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572522.747393:0:110139:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572522.747409:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.747437:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.747451:0:110139:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.747459:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572522.747464:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572522.747468:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:2.0:1788572522.747475:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383b5bbf8 00010000:00010000:2.0:1788572522.747479:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.747488:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572522.747502:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.747508:0:110139:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:2.0:1788572522.747512:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.747516:0:110139:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, flags=265 00000080:00200000:2.0:1788572522.747543:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), inode mode 81a4 mask 44 00000080:00200008:2.0:1788572522.747551:0:110139:0:(file.c:1090:ll_file_open()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), flags 154400 00000002:00010000:2.0:1788572522.747568:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0104240114401 00010000:00010000:2.0:1788572522.747610:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.747618:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572522.747628:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572522.747635:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ce00 x1875449863879936/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572522.747645:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.747663:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879936:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.747864:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.750982:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863879936:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.751117:0:110139:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.751128:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.751140:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.751152:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 4/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.751168:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc4617ce00 x1875449863879936/t4294983722(4294983722) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572522.751243:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ce00 x1875449863879936/t4294983722(4294983722) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.751258:0:110139:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 240 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572522.751289:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.751300:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 16 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572522.751307:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572522.751352:0:110139:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00200000:2.0:1788572522.751364:0:110139:0:(file.c:6843:ll_layout_conf()) [0x20000041d:0x2:0x0]: layout version change: 2 -> 2 00000080:00010000:2.0:1788572522.751369:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3526ee 00000080:00010000:2.0:1788572522.751375:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526ee from it ffff9e0383b5bbf0 00010000:00010000:2.0:1788572522.751380:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 3/1,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.751389:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572522.751419:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ce00 x1875449863879936/t4294983722(4294983722) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572522.751588:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), inode mode 81a4 mask 4 00000080:00200000:2.0:1788572522.751596:0:110139:0:(xattr.c:744:ll_xattr_get()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), xattr lov 00000080:00200008:2.0:1788572522.751619:0:110139:0:(file.c:6296:ll_getattr_dentry()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572522.751626:0:110139:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08),name=crhIBIW0-TeUOeG6ka7IzA 00000002:00010000:2.0:1788572522.751675:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.751686:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572522.751697:0:110139:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572522.751704:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572522.751710:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) 00000080:00010000:2.0:1788572522.751714:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3526e7 00000080:00010000:2.0:1788572522.751718:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526e7 from it ffff9e0383b5bcd8 00010000:00010000:2.0:1788572522.755097:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572522.755117:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200008:2.0:1788572522.755985:0:110139:0:(file.c:479:ll_file_release()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), flags 154000 00010000:00010000:2.0:1788572522.756169:0:110139:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572522.756472:0:110139:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572522.756493:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.756506:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.756516:0:110139:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572522.756524:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572522.756531:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572522.756534:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:2.0:1788572522.756540:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383b5bbf8 00010000:00010000:2.0:1788572522.756545:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.756553:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572522.756568:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.756575:0:110139:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=25 00000080:00200000:2.0:1788572522.756580:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.756585:0:110139:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, flags=265 00000080:00200000:2.0:1788572522.756620:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08), inode mode 81a4 mask 46 00000080:00200008:2.0:1788572522.756629:0:110139:0:(file.c:1090:ll_file_open()) START file crhIBIW0-TeUOeG6ka7IzA:[0x20000041d:0x2:0x0](ffff92fc665e4b08), flags 154402 00000002:00010000:2.0:1788572522.756650:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0104240114403 00010000:00010000:2.0:1788572522.756699:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.756708:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572522.756747:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572522.756759:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e680 x1875449863880192/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572522.756782:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.756805:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880192:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.756973:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.760424:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572522.760473:0:110079:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881872896 00000100:00100000:0.0:1788572522.760482:0:110079:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881872896 00000100:00100000:0.0:1788572522.760530:0:110079:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:0.0:1788572522.760540:0:110079:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6704b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:b4a32cca-995b-4bf1-a799-13b256fbfebe+5:6528:x1875449881872896:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572522.760559:0:110079:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366600/0x58a604dcef3526e7 lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792803 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572522.760637:0:110079:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572522.760644:0:110079:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6704b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:b4a32cca-995b-4bf1-a799-13b256fbfebe+5:6528:x1875449881872896:12345-192.168.202.132@tcp:104: Request processed in 105us (230us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572522.760651:0:110079:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:2.0:1788572522.769825:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880192:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.769859:0:110139:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode CW ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769876:0:110139:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 4/1,0 mode: --/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769883:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 4/1,0 mode: --/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769895:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 4/1,0 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769908:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 4/1,0 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769921:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 5/1,1 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769929:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 5/1,1 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769937:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 4/0,1 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.769948:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc4617e680 x1875449863880192/t4294983723(4294983723) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572522.769961:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e680 x1875449863880192/t4294983723(4294983723) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.769978:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 3/0,1 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.769989:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 8 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572522.769995:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572522.770317:0:110139:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x2:0x0], updating i_size 9000 00000080:00010000:2.0:1788572522.770324:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3526f5 00000080:00010000:2.0:1788572522.770331:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526f5 from it ffff9e0383b5bbf0 00010000:00010000:2.0:1788572522.770335:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 3/0,1 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.770345:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 2/0,0 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572522.770372:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e680 x1875449863880192/t4294983723(4294983723) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572522.770551:0:110139:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) cmd=800466af arg=7ffdbde6241c 00000080:00200080:2.0:1788572522.770566:0:110139:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) cmd=80086601 arg=7ffdbde623b8 00000100:00100000:2.0:1788572522.770608:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880448:192.168.202.132@tcp:33:lfs.0 00000100:00100000:2.0:1788572522.770786:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.774159:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880448:192.168.202.132@tcp:33:lfs.0 00000080:00200080:2.0:1788572522.774272:0:110139:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) cmd=c00c66f3 arg=7ffdbde623a4 00000002:00010000:2.0:1788572522.774313:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1788572522.774332:0:110139:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 2/0,0 mode: CW/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.774366:0:110139:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1788572522.774410:0:110139:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617ce00 x1875449863879936/t4294983722(4294983722) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572522.774474:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572522.774481:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e300 x1875449863880704/t0(0) o35->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572522.774503:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880704:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572522.774613:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.777701:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880704:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572522.777857:0:110139:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617ce00 x1875449863879936/t4294983722(4294983722) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572522.777903:0:110139:0:(vvp_object.c:117:vvp_conf_set()) [0x20000041d:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572522.777973:0:110139:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 1/0,0 mode: --/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.777983:0:110139:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572522.777990:0:110139:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788572522.777994:0:110139:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366a00/0x58a604dcef3526f5 lrc: 0/0,0 mode: --/CW res: [0x20000041d:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179282d expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.778133:0:110139:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367200/0x58a604dcef3526ee lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0xd/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792818 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572522.778237:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.778245:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572522.780429:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572522.780442:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e300 x1875449863880960/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 768/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572522.780454:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x81000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.780484:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880960:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.780649:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.784221:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863880960:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.784250:0:110139:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784262:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 4/1,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784285:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784298:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784317:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784325:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784332:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 4/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.784344:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff92fc4617e300 x1875449863880960/t4294983725(4294983725) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 768/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572522.784358:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e300 x1875449863880960/t4294983725(4294983725) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 768/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.784374:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784389:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 2 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572522.784395:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1788572522.784416:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617e300 x1875449863880960/t4294983725(4294983725) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 768/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572522.784426:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3526fc 00000080:00010000:2.0:1788572522.784433:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526fc from it ffff9e0383b5bcc0 00010000:00010000:2.0:1788572522.784436:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784449:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1788572522.784509:0:110139:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) cmd=c00c66f3 arg=2082530 00010000:00010000:2.0:1788572522.784563:0:110139:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.784577:0:110139:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef3526fc lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0xc9a239a221792842 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.784602:0:110139:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617e300 x1875449863880960/t4294983725(4294983725) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 768/848 e 0 to 0 dl 1788572538 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572522.784613:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572522.784618:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ce00 x1875449863881216/t0(0) o35->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:23/10 lens 504/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572522.784642:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863881216:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572522.784717:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.797657:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863881216:192.168.202.132@tcp:35:lfs.0 00000080:00020000:2.0:1788572522.797680:0:110139:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff92fc5824a800: inode [0x20000041d:0x2:0x0] mdc close failed: rc = -22 00000080:00200000:2.0:1788572522.840364:0:110139:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000420:0x1:0x0](ffff92fc665e3a88) 00000080:00200000:2.0:1788572522.840374:0:110139:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000420:0x1:0x0], updating i_size 0 00000080:00200000:2.0:1788572522.840408:0:110139:0:(file.c:6843:ll_layout_conf()) [0x200000420:0x1:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1788572522.840413:0:110139:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000420:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572522.840416:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000420:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef35270a 00000080:00010000:2.0:1788572522.840888:0:110139:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000420:0x1:0x0](ffff92fc665e3a88) 00000080:00200008:2.0:1788572522.840906:0:110139:0:(file.c:1090:ll_file_open()) START file . :VOLATILE:0000:3801E26C:fd=03:[0x200000420:0x1:0x0](ffff92fc665e3a88), flags 574702 00000002:00100000:2.0:1788572522.840954:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67339500 x1875449863881728/t4294983727(4294983727) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 584/640 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572522.840987:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35270a from it ffff92fc48215120 00010000:00010000:2.0:1788572522.840992:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 3/1,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.841003:0:110139:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 2/0,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.841035:0:110139:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000420:0x1:0x0](ffff92fc665e3a88), inode mode 8180 mask 40 00000080:00200080:2.0:1788572522.841059:0:110139:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) cmd=c00c66f3 arg=7ffdbde623a4 00000002:00010000:2.0:1788572522.841077:0:110139:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x2:0x0]) in obj [0x20000041d:0x2:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1788572522.841126:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.841134:0:110139:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572522.841146:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572522.841153:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c000 x1875449863881984/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572522.841161:0:110139:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x81000000000000 pid: 110139 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.841193:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863881984:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.841373:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.844246:0:110139:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863881984:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.844272:0:110139:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844294:0:110139:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 4/1,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844306:0:110139:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844317:0:110139:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 4/1,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844329:0:110139:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844337:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 5/1,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844345:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 4/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.844356:0:110139:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff92fc4617c000 x1875449863881984/t4294983728(4294983728) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572522.844369:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c000 x1875449863881984/t4294983728(4294983728) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572522.844385:0:110139:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844399:0:110139:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 2 res 8589935645/2 (0 0) 00000080:00200000:2.0:1788572522.844406:0:110139:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x2:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1788572522.844426:0:110139:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c000 x1875449863881984/t4294983728(4294983728) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572522.844435:0:110139:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x2:0x0](ffff92fc665e4b08) for lock 0x58a604dcef352711 00000080:00010000:2.0:1788572522.844441:0:110139:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352711 from it ffff9e0383b5bcc0 00010000:00010000:2.0:1788572522.844444:0:110139:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 3/0,1 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844451:0:110139:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200080:2.0:1788572522.844510:0:110139:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) cmd=c00c66f3 arg=2082550 00010000:00010000:2.0:1788572522.844559:0:110139:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 2/0,0 mode: PW/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.844573:0:110139:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352711 lrc: 0/0,0 mode: --/PW res: [0x20000041d:0x2:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0xc9a239a22179287a expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.844587:0:110139:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617c000 x1875449863881984/t4294983728(4294983728) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/848 e 0 to 0 dl 1788572538 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572522.844597:0:110139:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572522.844602:0:110139:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d500 x1875449863882240/t0(0) o35->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:23/10 lens 504/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572522.844623:0:110139:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110139:1875449863882240:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572522.844694:0:110139:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.847694:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572522.847876:0:110079:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881873024 00000100:00100000:1.0:1788572522.847886:0:110079:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881873024 00000100:00100000:1.0:1788572522.847937:0:110079:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 1 00000100:00100000:1.0:1788572522.847947:0:110079:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc6704b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:b4a32cca-995b-4bf1-a799-13b256fbfebe+5:6532:x1875449881873024:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572522.847964:0:110079:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572522.848742:0:110079:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572522.848754:0:110079:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc6704b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:b4a32cca-995b-4bf1-a799-13b256fbfebe+5:6532:x1875449881873024:12345-192.168.202.132@tcp:104: Request processed in 806us (1066us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572522.848762:0:110079:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 1 00010000:00010000:2.0:1788572522.848849:0:110090:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572522.848857:0:110090:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.848867:0:110090:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205800 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572522.848873:0:110090:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.848881:0:110090:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.848890:0:110090:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 3/0,0 mode: CR/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572522.848914:0:110090:0:(vvp_object.c:117:vvp_conf_set()) [0x20000041d:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572522.848962:0:110090:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 2/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.848969:0:110090:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572522.848998:0:110090:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.849321:0:110090:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205800/0x58a604dcef352703 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x2:0x0].0x0 bits 0x8/0x8 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a221792857 expref: -99 pid: 110139 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572522.849350:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572522.849352:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572522.849354:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572522.853461:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572522.853551:0:110080:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881873152 00000100:00100000:1.0:1788572522.853561:0:110080:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881873152 00000100:00100000:1.0:1788572522.853616:0:110080:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 2 00000100:00100000:1.0:1788572522.853627:0:110080:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67049180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:b4a32cca-995b-4bf1-a799-13b256fbfebe+5:6532:x1875449881873152:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572522.853646:0:110080:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 2/0,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572522.853747:0:110080:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572522.853754:0:110080:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67049180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:b4a32cca-995b-4bf1-a799-13b256fbfebe+5:6532:x1875449881873152:12345-192.168.202.132@tcp:104: Request processed in 130us (304us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572522.853762:0:110080:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 2 00010000:00010000:2.0:1788572522.854805:0:110089:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572522.854815:0:110089:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 2/0,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.854826:0:110089:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572522.854832:0:110089:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 2/0,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.854840:0:110089:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 2/0,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.854850:0:110089:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 3/0,0 mode: PR/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572522.854873:0:110089:0:(vvp_object.c:117:vvp_conf_set()) [0x200000420:0x1:0x0]: losing layout lock 00010000:00010000:2.0:1788572522.854917:0:110089:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 2/0,0 mode: --/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.854924:0:110089:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572522.854942:0:110089:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 1/0,0 mode: --/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.854950:0:110089:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70205400/0x58a604dcef35270a lrc: 0/0,0 mode: --/PR res: [0x200000420:0x1:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xc9a239a221792865 expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572522.854977:0:110089:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572522.854978:0:110089:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572522.854980:0:110089:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572522.854995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b4a32cca-995b-4bf1-a799-13b256fbfebe:2396:1875449863882496:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572522.864576:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b4a32cca-995b-4bf1-a799-13b256fbfebe:2396:1875449863882496:192.168.202.132@tcp:103:ldlm_bl.0 00000080:00200000:0.0:1788572522.887594:0:110140:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:0.0:1788572522.887631:0:110140:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572522.887654:0:110140:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572522.887670:0:110140:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572522.887679:0:110140:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572522.887686:0:110140:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:0.0:1788572522.887690:0:110140:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:0.0:1788572522.887699:0:110140:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e53bf8 00010000:00010000:0.0:1788572522.887703:0:110140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572522.887712:0:110140:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572522.887944:0:110140:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572522.887960:0:110140:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665d9988), flags=27 00000080:00200000:0.0:1788572522.887966:0:110140:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665d9988), intent=0 00000002:00010000:0.0:1788572522.887980:0:110140:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572522.888146:0:110140:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.888155:0:110140:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.888168:0:110140:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572522.888193:0:110140:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883264:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572522.888345:0:110140:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.895317:0:110140:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883264:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572522.895348:0:110140:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895370:0:110140:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895384:0:110140:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895395:0:110140:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895406:0:110140:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895420:0:110140:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895428:0:110140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895435:0:110140:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572522.895447:0:110140:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc6704bb80 x1875449863883264/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572522.895460:0:110140:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895470:0:110140:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc498e4c00 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:0.0:1788572522.895477:0:110140:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572522.895498:0:110140:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00200000:0.0:1788572522.895502:0:110140:0:(namei.c:147:ll_iget()) got inode: [0x20000041d:0x1:0x0](ffff92fc665e0908): rc = 0 00000080:00010000:0.0:1788572522.895513:0:110140:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef35271f 00000080:00010000:0.0:1788572522.895526:0:110140:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e0908) 00000080:00010000:0.0:1788572522.895529:0:110140:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35271f from it ffff9e0383e53ac8 00010000:00010000:0.0:1788572522.895532:0:110140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895542:0:110140:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc2a00/0x58a604dcef35271f lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217928b9 expref: -99 pid: 110140 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572522.895573:0:110140:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:0.0:1788572522.895593:0:110140:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), file ffff92fc45814180, open_flags 18800, mode 0 opened 0 00000080:00200000:0.0:1788572522.895599:0:110140:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), intent=open 00000002:00010000:0.0:1788572522.895656:0:110140:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:0.0:1788572522.895703:0:110140:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.895711:0:110140:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572522.895965:0:110140:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572522.895973:0:110140:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704bb80 x1875449863883520/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572522.895982:0:110140:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572522.896003:0:110140:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883520:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572522.896428:0:110140:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.899658:0:110140:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883520:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572522.899683:0:110140:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.899697:0:110140:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.899704:0:110140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.899711:0:110140:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.901888:0:110140:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.901904:0:110140:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3e00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572522.901909:0:110140:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.901917:0:110140:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.901926:0:110140:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.901933:0:110140:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572522.901941:0:110140:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.901948:0:110140:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3e00/0x58a604dcef352726 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572522.901980:0:110140:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc6704bb80 x1875449863883520/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572522.901994:0:110140:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:0.0:1788572522.902435:0:110140:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:0.0:1788572522.902457:0:110140:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572522.902472:0:110140:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572522.902485:0:110140:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572522.902493:0:110140:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572522.902498:0:110140:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:0.0:1788572522.902501:0:110140:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:0.0:1788572522.902506:0:110140:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e53bf8 00010000:00010000:0.0:1788572522.902510:0:110140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572522.902519:0:110140:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572522.902533:0:110140:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572522.902540:0:110140:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:0.0:1788572522.902547:0:110140:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 44 00000080:00200008:0.0:1788572522.902556:0:110140:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e0908), flags 304000 00000002:00010000:0.0:1788572522.902569:0:110140:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20000041d:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788572522.902613:0:110140:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.902619:0:110140:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572522.902630:0:110140:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572522.902638:0:110140:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704bb80 x1875449863883776/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572522.902647:0:110140:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572522.902669:0:110140:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883776:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572522.909452:0:110140:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.910569:0:110140:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883776:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572522.910592:0:110140:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910608:0:110140:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910616:0:110140:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910623:0:110140:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910632:0:110140:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910641:0:110140:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572522.910645:0:110140:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910652:0:110140:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910658:0:110140:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910665:0:110140:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572522.910671:0:110140:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.910678:0:110140:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc51bc3200/0x58a604dcef35272d lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 110140 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572522.910697:0:110140:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc6704bb80 x1875449863883776/t4294983735(4294983735) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572522.910711:0:110140:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6704bb80 x1875449863883776/t4294983735(4294983735) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572522.910890:0:110140:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572522.910913:0:110140:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000002:00100000:0.0:1788572522.910934:0:110140:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6704bb80 x1875449863883776/t4294983735(4294983735) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572538 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:0.0:1788572522.911099:0:110140:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908) cmd=c0086915 arg=13a5620 00000100:00100000:0.0:1788572522.911194:0:110140:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883904:192.168.202.132@tcp:34:lfs.0 00000100:00100000:0.0:1788572522.911322:0:110140:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.914298:0:110140:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863883904:192.168.202.132@tcp:34:lfs.0 00000080:00200008:0.0:1788572522.916205:0:110140:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e0908), flags 304000 00000002:00100000:0.0:1788572522.916298:0:110140:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6704bb80 x1875449863883776/t4294983735(4294983735) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572538 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572522.916311:0:110140:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572522.916316:0:110140:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704ad80 x1875449863884160/t0(0) o35->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572522.916339:0:110140:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863884160:192.168.202.132@tcp:35:lfs.0 00000100:00100000:0.0:1788572522.916410:0:110140:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.919337:0:110140:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110140:1875449863884160:192.168.202.132@tcp:35:lfs.0 00000002:00100000:0.0:1788572522.919373:0:110140:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6704bb80 x1875449863883776/t4294983735(4294983735) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572522.930325:0:110141:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572522.930352:0:110141:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.930371:0:110141:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572522.930387:0:110141:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572522.930396:0:110141:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572522.930404:0:110141:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572522.930407:0:110141:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:2.0:1788572522.930414:0:110141:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e5bbf8 00010000:00010000:2.0:1788572522.930418:0:110141:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572522.930427:0:110141:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572522.930442:0:110141:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.930449:0:110141:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:2.0:1788572522.930455:0:110141:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.930466:0:110141:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), file ffff92fc4a133500, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572522.930472:0:110141:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), intent=open 00000002:00010000:2.0:1788572522.930524:0:110141:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572522.930579:0:110141:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352734 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110141 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.930587:0:110141:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352734 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110141 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572522.930599:0:110141:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572522.930612:0:110141:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ca80 x1875449863884416/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572522.930621:0:110141:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70367c00/0x58a604dcef352734 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110141 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572522.930642:0:110141:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110141:1875449863884416:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572522.930999:0:110141:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37f00 going to sleep for 11 seconds 00000080:00200000:1.0:1788572522.960924:0:110142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:1.0:1788572522.960962:0:110142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572522.960986:0:110142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572522.961004:0:110142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572522.961137:0:110142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572522.961346:0:110142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:1.0:1788572522.961350:0:110142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:1.0:1788572522.961359:0:110142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e63bf8 00010000:00010000:1.0:1788572522.961364:0:110142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572522.961374:0:110142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572522.961392:0:110142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572522.961402:0:110142:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:1.0:1788572522.961409:0:110142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:1.0:1788572522.961422:0:110142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), file ffff92fc608be600, open_flags 18800, mode 0 opened 0 00000080:00200000:1.0:1788572522.961428:0:110142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), intent=open 00000002:00010000:1.0:1788572522.961487:0:110142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:1.0:1788572522.961548:0:110142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572522.961557:0:110142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572522.961573:0:110142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572522.961586:0:110142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6728b480 x1875449863885056/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572522.961596:0:110142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572522.961619:0:110142:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110142:1875449863885056:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572522.962265:0:110142:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.965751:0:110142:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110142:1875449863885056:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572522.965782:0:110142:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965797:0:110142:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965803:0:110142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965811:0:110142:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965821:0:110142:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965827:0:110142:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f4800 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572522.965832:0:110142:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965839:0:110142:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965846:0:110142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965854:0:110142:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572522.965860:0:110142:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572522.965867:0:110142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f4800/0x58a604dcef352742 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 110142 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572522.965886:0:110142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc6728b480 x1875449863885056/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572522.965902:0:110142:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:2.0:1788572522.966041:0:110142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572522.966063:0:110142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572522.966078:0:110142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572522.966091:0:110142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572522.966099:0:110142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572522.966104:0:110142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572522.966107:0:110142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:2.0:1788572522.966112:0:110142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e63bf8 00010000:00010000:2.0:1788572522.966116:0:110142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572522.966125:0:110142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572522.966142:0:110142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572522.966151:0:110142:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:2.0:1788572522.966158:0:110142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 44 00000080:00200008:2.0:1788572522.966166:0:110142:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e0908), flags 304000 00000080:00200080:2.0:1788572522.966205:0:110142:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908) cmd=c0086915 arg=1f4d620 00000100:00100000:2.0:1788572522.966303:0:110142:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110142:1875449863885184:192.168.202.132@tcp:34:lfs.0 00000100:00100000:2.0:1788572522.966382:0:110142:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b37540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572522.969361:0:110142:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110142:1875449863885184:192.168.202.132@tcp:34:lfs.0 00000080:00200008:2.0:1788572522.969427:0:110142:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e0908), flags 304000 00010000:00010000:2.0:1788572522.969461:0:110142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef35273b lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792914 expref: -99 pid: 110141 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572522.983072:0:110143:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:1.0:1788572522.983109:0:110143:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572522.983131:0:110143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572522.983148:0:110143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572522.983156:0:110143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572522.983164:0:110143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:1.0:1788572522.983168:0:110143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:1.0:1788572522.983177:0:110143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e33bf8 00010000:00010000:1.0:1788572522.983181:0:110143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572522.983189:0:110143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572522.983206:0:110143:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572522.983213:0:110143:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=27 00000080:00200000:1.0:1788572522.983220:0:110143:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:1.0:1788572522.983234:0:110143:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), file ffff92fc47e64780, open_flags 18800, mode 0 opened 0 00000080:00200000:1.0:1788572522.983241:0:110143:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=crhIBIW0-TeUOeG6ka7IzA, dir=[0x20000041d:0x1:0x0](ffff92fc665e0908), intent=open 00000002:00010000:1.0:1788572522.983312:0:110143:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: r¸H…´ù7”9ẑ®ÈÌ,[0x0:0x0:0x0]) in obj [0x20000041d:0x1:0x0], intent: open flags 0304001 00010000:00010000:1.0:1788572522.983364:0:110143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572522.983372:0:110143:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572522.983383:0:110143:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc5824a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572522.983394:0:110143:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67289f80 x1875449863885440/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572522.983403:0:110143:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 3/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572522.983435:0:110143:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110143:1875449863885440:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572522.983604:0:110143:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b59c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.987290:0:110143:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110143:1875449863885440:192.168.202.132@tcp:101:lfs.0 00010000:00010000:0.0:1788572522.987318:0:110143:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987331:0:110143:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987338:0:110143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 4/1,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987345:0:110143:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987359:0:110143:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987366:0:110143:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f5c00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572522.987371:0:110143:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987379:0:110143:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 4/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987387:0:110143:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 5/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987395:0:110143:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572522.987402:0:110143:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 3/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572522.987408:0:110143:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc702f5c00/0x58a604dcef352749 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 110143 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572522.987426:0:110143:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff92fc67289f80 x1875449863885440/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5824a800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572538 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572522.987439:0:110143:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x20000041d:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:0.0:1788572522.987587:0:110143:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:0.0:1788572522.987609:0:110143:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572522.987623:0:110143:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788572522.987635:0:110143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788572522.987643:0:110143:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572522.987650:0:110143:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:0.0:1788572522.987653:0:110143:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef3526d9 00000080:00010000:0.0:1788572522.987659:0:110143:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3526d9 from it ffff9e0383e33bf8 00010000:00010000:0.0:1788572522.987662:0:110143:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788572522.987670:0:110143:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70366200/0x58a604dcef3526d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217927cb expref: -99 pid: 110139 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572522.987685:0:110143:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572522.987693:0:110143:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59b.sanity-sec, flags=267 00000080:00200000:0.0:1788572522.987700:0:110143:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 44 00000080:00200008:0.0:1788572522.987708:0:110143:0:(file.c:1090:ll_file_open()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e0908), flags 304000 00000080:00200080:0.0:1788572522.987788:0:110143:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908) cmd=c0086915 arg=1c03620 00000100:00100000:0.0:1788572522.987872:0:110143:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110143:1875449863885568:192.168.202.132@tcp:34:lfs.0 00000100:00100000:0.0:1788572522.987943:0:110143:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48d3bf00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572522.991594:0:110143:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job lfs:b4a32cca-995b-4bf1-a799-13b256fbfebe:110143:1875449863885568:192.168.202.132@tcp:34:lfs.0 00000080:00200008:0.0:1788572522.991792:0:110143:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e0908), flags 304000 00010000:00010000:0.0:1788572522.991832:0:110143:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef35273b lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792914 expref: -99 pid: 110141 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572523.081954:0:110147:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:0.0:1788572523.081987:0:110147:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc5824a800 00010000:00010000:0.0:1788572523.082023:0:110147:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/4 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788572523.082049:0:110149:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc5824a800 00010000:00010000:0.0:1788572523.082068:0:110149:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc5824a800 00010000:00010000:0.0:1788572523.082085:0:110149:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc5824a800 00000080:00200000:2.0:1788572523.172197:0:109387:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e0908) 00000080:00200000:2.0:1788572523.172244:0:109387:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572523.172257:0:109387:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572523.172326:0:109387:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572523.172333:0:109387:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665e4b08) 00000080:00200000:2.0:1788572523.172424:0:109387:0:(vvp_io.c:1826:vvp_io_init()) [0x200000420:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572523.172430:0:109387:0:(vvp_io.c:304:vvp_io_fini()) [0x200000420:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572523.172433:0:109387:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000420:0x1:0x0](ffff92fc665e3a88) 00010000:00010000:2.0:1788572523.525603:0:110089:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572525.572914:0:110089:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572526.554534:0:110179:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572526.554545:0:110179:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572526.554563:0:110179:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572526.554579:0:110179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572526.554592:0:110179:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572526.554598:0:110179:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572526.554604:0:110179:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572526.554606:0:110179:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef352750 00000080:00010000:2.0:1788572526.554613:0:110179:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352750 from it ffff9e0383ee3cb8 00010000:00010000:2.0:1788572526.554616:0:110179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572526.554623:0:110179:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788572526.554757:0:110179:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572526.554762:0:110179:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988),name=/ 00000002:00010000:2.0:1788572526.554771:0:110179:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572526.554778:0:110179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572526.554786:0:110179:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572526.554792:0:110179:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572526.554796:0:110179:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00010000:2.0:1788572526.554799:0:110179:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d9988) for lock 0x58a604dcef352750 00000080:00010000:2.0:1788572526.554803:0:110179:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352750 from it ffff9e0383ee3cb8 00010000:00010000:2.0:1788572526.554806:0:110179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572526.554813:0:110179:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5824a800 lock: ffff92fc70204800/0x58a604dcef352750 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792992 expref: -99 pid: 110171 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572526.567316:0:110179:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d9988) 00000080:00200000:1.0:1788572526.567854:0:110179:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc5824a800 (sb ffff92fc5824a800) 00010000:00010000:1.0:1788572526.567898:0:110179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567911:0:110179:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567919:0:110179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567927:0:110179:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc702f5600/0x58a604dcef352670 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792793 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567951:0:110179:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572526.567958:0:110179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567964:0:110179:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567971:0:110179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567976:0:110179:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2000/0x58a604dcef3526a1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179279a expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.567993:0:110179:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1788572526.568008:0:110179:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217927a1 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.568014:0:110179:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217927a1 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.568020:0:110179:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217927a1 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.568026:0:110179:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367800/0x58a604dcef3526a8 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217927a1 expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572526.568034:0:110179:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:1.0:1788572526.568038:0:110179:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:1.0:1788572526.573595:0:110179:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc58336800 active: 1 00020000:01000000:1.0:1788572526.573606:0:110179:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc58332000 active: 1 00020000:01000000:1.0:1788572526.573609:0:110179:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:1.0:1788572526.573614:0:110179:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc5824a800 00020000:01000000:1.0:1788572526.573617:0:110179:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc5824a800: disconnecting target lustre-OST0001-osc-ffff92fc5824a800 00000100:00080000:1.0:1788572526.573635:0:110179:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import b4a32cca-995b-4bf1-a799-13b256fbfebe->lustre-OST0001_UUID 00000100:00080000:1.0:1788572526.573756:0:110179:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572526.573800:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863886336:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572526.588072:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572526.588094:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863886336:192.168.202.132@tcp:9:umount.0 00000020:00000080:1.0:1788572526.589481:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58332000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc5824a800 00000020:00000080:0.0:1788572526.591452:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51d2d000 for lustre-OST0001-osc-ffff92fc5824a800 00000020:00000080:0.0:1788572526.591466:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc51d2f000/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:0.0:1788572526.591470:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2f000/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-OST0001-osc-ffff92fc5824a800 00000020:01000000:0.0:1788572526.591482:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc5824a800 (b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000100:00100000:1.0:1788572526.592460:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863886720:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572526.596123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2a800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572526.596154:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863886720:192.168.202.132@tcp:9:umount.0 00000020:00000080:0.0:1788572526.596523:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58336800/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc5824a800 00010000:00010000:2.0:1788572526.596982:0:110090:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000080:0.0:1788572526.597412:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51d2a800 for lustre-OST0000-osc-ffff92fc5824a800 00000020:00000080:0.0:1788572526.597645:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6090c800/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-clilov-ffff92fc5824a800 00000100:00100000:1.0:1788572526.599487:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863886976:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572526.602652:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58c8a000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572526.602679:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863886976:192.168.202.132@tcp:39:umount.0 00000100:00100000:0.0:1788572526.602973:0:110179:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572526.607719:0:110179:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3526b6 00000020:00000080:0.0:1788572526.607773:0:110179:0:(genops.c:885:class_export_put()) final put ffff92fc51d2b000/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000100:00080000:0.0:1788572526.607874:0:110179:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import b4a32cca-995b-4bf1-a799-13b256fbfebe->lustre-MDT0001_UUID 00000020:00000080:1.0:1788572526.608107:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2b000/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-MDT0000-mdc-ffff92fc5824a800 00000100:00080000:0.0:1788572526.616042:0:110179:0:(import.c:65:import_set_state_nolock()) ffff92fc492e1800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572526.616632:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863887104:192.168.202.132@tcp:39:umount.0 00000100:00100000:2.0:1788572526.627552:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5824a800: committing for last_committed 4294976749 gen 1 00000100:00080000:2.0:1788572526.627566:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc492e1800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572526.627628:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b4a32cca-995b-4bf1-a799-13b256fbfebe:2392:1875449863887104:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572526.627686:0:110179:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572526.627689:0:110179:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5824a800: committing for last_committed 4294976749 gen 2 00000100:00100000:0.0:1788572526.627694:0:110179:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572526.632569:0:110179:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3526bd 00000020:00000080:0.0:1788572526.632587:0:110179:0:(genops.c:885:class_export_put()) final put ffff92fc51d28000/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:0.0:1788572526.632718:0:110179:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3526af 00000020:00000080:0.0:1788572526.632763:0:110179:0:(genops.c:885:class_export_put()) final put ffff92fc51d2c800/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:1.0:1788572526.632918:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d28000/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-MDT0001-mdc-ffff92fc5824a800 00000020:01000000:0.0:1788572526.633127:0:110179:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc5824a800 (flags='') 00000020:00000080:0.0:1788572526.633138:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572526.633145:0:110179:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc5824a800: forcing exports to disconnect: 0/1 00000020:00080000:0.0:1788572526.633151:0:110179:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc51cd8000) has no exports 00000020:00000080:0.0:1788572526.633228:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572526.633235:0:110179:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc5824a800 (uuid b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000020:00000080:0.0:1788572526.633239:0:110179:0:(genops.c:885:class_export_put()) final put ffff92fc46155000/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:0.0:1788572526.633242:0:110179:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46155000/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-clilov-ffff92fc5824a800 00000020:01000000:0.0:1788572526.633246:0:110179:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc5824a800 (b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000020:01000000:0.0:1788572526.633422:0:110179:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc5824a800 (flags='') 00000020:00000080:0.0:1788572526.633426:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572526.633430:0:110179:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc5824a800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572526.633432:0:110179:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc51cdcd40) has no exports 00000020:00000080:0.0:1788572526.633487:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572526.633491:0:110179:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc5824a800 (uuid b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000020:01000000:0.0:1788572526.633495:0:110179:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc5824a800 (flags='') 00000020:00000080:0.0:1788572526.633498:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572526.633501:0:110179:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc5824a800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572526.633503:0:110179:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc51cdb380) has no exports 00000020:00000080:1.0:1788572526.634657:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2c800/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-clilmv-ffff92fc5824a800 00000020:00000080:1.0:1788572526.634669:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc46150000/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:1.0:1788572526.634671:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc46150000/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-clilmv-ffff92fc5824a800 00000020:01000000:1.0:1788572526.634678:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc5824a800 (b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000008:01000000:1.0:1788572526.635521:0:110179:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc5824a800: client import never connected 00000100:00080000:1.0:1788572526.635529:0:110179:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572526.635533:0:110179:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5824a800: committing for last_committed 4294983720 gen 3 00000100:00100000:1.0:1788572526.635542:0:110179:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572526.653496:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572526.653510:0:110179:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc5824a800 (uuid b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000020:01000000:0.0:1788572526.653516:0:110179:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc5824a800 (flags='') 00000020:00000080:0.0:1788572526.653521:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572526.653524:0:110179:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc5824a800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572526.653529:0:110179:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc51cd99c0) has no exports 00000020:00000080:1.0:1788572526.654181:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc58c8a000 for lustre-MDT0000-mdc-ffff92fc5824a800 00000020:00000080:1.0:1788572526.654190:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc58c8d800/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:1.0:1788572526.654193:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58c8d800/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-MDT0000-mdc-ffff92fc5824a800 00000020:01000000:1.0:1788572526.654214:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc5824a800 (b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000008:01000000:0.0:1788572526.655406:0:110179:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc5824a800: client import never connected 00000100:00080000:0.0:1788572526.655414:0:110179:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572526.655418:0:110179:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5824a800: committing for last_committed 4294976749 gen 3 00000100:00100000:0.0:1788572526.655427:0:110179:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572526.673440:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572526.673455:0:110179:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc5824a800 (uuid b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000080:02000400:0.0:1788572526.673460:0:110179:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:0.0:1788572526.673463:0:110179:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:0.0:1788572526.673508:0:110179:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc5824a800 00000100:00080000:0.0:1788572526.673516:0:110179:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 124343f7-953d-45b3-8c00-3a0ab68d71b0->MGS 00010000:00010000:0.0:1788572526.673530:0:110179:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572526.673543:0:110179:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572526.673550:0:110179:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572526.673553:0:110179:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:0.0:1788572526.673556:0:110179:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:0.0:1788572526.673598:0:110179:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572526.673604:0:110179:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572526.673621:0:110179:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53ea2d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788572526.673633:0:110179:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job umount:124343f7-953d-45b3-8c00-3a0ab68d71b0:110179:1875449863887232:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572526.673643:0:110179:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572526.673650:0:110179:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572526.673653:0:110179:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f4200/0x58a604dcef352669 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179278c expref: -99 pid: 110078 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572526.673679:0:110179:0:(import.c:65:import_set_state_nolock()) ffff92fc67141800 MGS: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1788572526.673704:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:124343f7-953d-45b3-8c00-3a0ab68d71b0:2392:1875449863887360:192.168.202.132@tcp:251:umount.0 00000020:00000080:2.0:1788572526.675751:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc492e1800 for lustre-MDT0001-mdc-ffff92fc5824a800 00000020:00000080:2.0:1788572526.675768:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc481c2800/b4a32cca-995b-4bf1-a799-13b256fbfebe 00000020:00000080:2.0:1788572526.675771:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc481c2800/b4a32cca-995b-4bf1-a799-13b256fbfebe for lustre-MDT0001-mdc-ffff92fc5824a800 00000020:01000000:2.0:1788572526.675788:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc5824a800 (b4a32cca-995b-4bf1-a799-13b256fbfebe) 00000100:00080000:0.0:1788572526.676881:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc67141800 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:0.0:1788572526.676890:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788572526.676894:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788572526.676895:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572526.677104:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:124343f7-953d-45b3-8c00-3a0ab68d71b0:2392:1875449863887360:192.168.202.132@tcp:251:umount.0 00000100:00080000:0.0:1788572526.677148:0:110179:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:0.0:1788572526.677151:0:110179:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572526.677154:0:110179:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788572526.677156:0:110179:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:0.0:1788572526.677166:0:110179:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352662 00000020:00000080:0.0:1788572526.677173:0:110179:0:(genops.c:885:class_export_put()) final put ffff92fc67140000/124343f7-953d-45b3-8c00-3a0ab68d71b0 00000020:01000000:0.0:1788572526.677183:0:110179:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:0.0:1788572526.677189:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572526.677332:0:110179:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572526.677337:0:110179:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc70034d40) has no exports 10000000:01000000:0.0:1788572526.677375:0:110092:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:0.0:1788572526.679833:0:110179:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:0.0:1788572526.681790:0:110179:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:0.0:1788572526.681795:0:110179:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572526.681800:0:110179:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788572526.681833:0:110179:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:0.0:1788572526.682903:0:110179:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572526.682917:0:110179:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 124343f7-953d-45b3-8c00-3a0ab68d71b0) 00000020:01000004:0.0:1788572526.682924:0:110179:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc5824a800 1 00000020:01000004:0.0:1788572526.682929:0:110179:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc5824b000 00000020:00000080:0.0:1788572526.707160:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67140000/124343f7-953d-45b3-8c00-3a0ab68d71b0 for MGC192.168.202.132@tcp 00000020:00000080:0.0:1788572526.707188:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc67141800 for MGC192.168.202.132@tcp 00000020:00000080:0.0:1788572526.707210:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc5824c000/124343f7-953d-45b3-8c00-3a0ab68d71b0 00000020:00000080:0.0:1788572526.707216:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824c000/124343f7-953d-45b3-8c00-3a0ab68d71b0 for MGC192.168.202.132@tcp 00000020:01000000:0.0:1788572526.707221:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (124343f7-953d-45b3-8c00-3a0ab68d71b0) 00010000:00010000:2.0:1788572526.725125:0:110090:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00000100:00100000:1.0:1788572526.725608:0:110080:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc463ec000:110080] 1 exiting: rc = 0 00000100:00100000:0.0:1788572526.726886:0:110079:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc51c3c800:110079] 0 exiting: rc = 1 00000100:00100000:2.0:1788572526.736565:0:110081:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc463ec400:110081] 0 exiting: rc = 1 00000100:00100000:0.0:1788572526.742473:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.742487:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.744154:0:110083:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc49953e00:110083] 0 exiting: rc = 1 00000100:00100000:0.0:1788572526.744270:0:110084:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc48dd2200:110084] 1 exiting: rc = 0 00000100:00100000:0.0:1788572526.744323:0:110085:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc48dd2800:110085] 2 exiting: rc = 0 00000100:00100000:2.0:1788572526.744763:0:110086:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc48dd3000:110086] 0 exiting: rc = 1 00000100:00100000:0.0:1788572526.746492:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746502:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746509:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746516:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746522:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746528:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746533:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746539:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746544:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746550:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746556:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746562:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746570:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746576:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746581:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746587:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746593:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746599:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746605:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746612:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746618:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746624:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746629:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746635:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746640:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746645:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746650:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746654:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746660:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746665:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746670:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746676:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746681:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746686:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746693:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746700:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746706:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746711:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746717:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746742:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.746748:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747199:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747208:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747213:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747219:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747224:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747230:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747236:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747242:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747247:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747253:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747269:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747275:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747280:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747286:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747290:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747295:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747300:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747305:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747309:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747314:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747320:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747325:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747330:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747338:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747343:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747348:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747354:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747360:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747366:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747372:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747377:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747383:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747392:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747399:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747407:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747413:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747418:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747423:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747429:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747436:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747442:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747782:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747789:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747796:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747802:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747808:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747814:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747820:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747826:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747833:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747839:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747846:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747852:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747858:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747865:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747872:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747881:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747889:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747899:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747907:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747915:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747933:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747940:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747947:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747955:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747962:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747972:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.747992:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748002:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748009:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748016:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748021:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748028:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748034:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748040:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748048:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748054:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748061:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748066:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748072:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748077:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748082:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748087:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748092:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748097:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748102:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572526.748107:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:1.0:1788572527.343894:0:110192:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:1.0:1788572527.343898:0:110192:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:1.0:1788572527.343900:0:110192:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:1.0:1788572527.344134:0:110192:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock,test_dummy_encryption 00000080:01200004:1.0:1788572527.344268:0:110192:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc67142000 00000080:01000004:1.0:1788572527.344278:0:110192:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:1.0:1788572527.344294:0:110192:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:1.0:1788572527.344398:0:110192:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:1.0:1788572527.344410:0:110192:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc49933380) 00000020:00000080:1.0:1788572527.344424:0:110192:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:1.0:1788572527.345317:0:110192:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572527.345409:0:110192:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572527.345553:0:110192:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:1.0:1788572527.345560:0:110192:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:1.0:1788572527.346673:0:110192:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:1.0:1788572527.346681:0:110192:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:1.0:1788572527.349675:0:110192:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572527.350059:0:110192:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572527.350251:0:110192:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572527.350268:0:110192:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:1.0:1788572527.352240:0:110192:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:1.0:1788572527.352248:0:110192:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00010000:00010000:1.0:1788572527.360537:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.360544:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.360546:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.361137:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.361139:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.361141:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572527.366607:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000080:2.0:1788572527.371017:0:110192:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid ea115449-6775-4569-9785-705ca0d938bc) 00000020:00000080:2.0:1788572527.371041:0:110192:0:(genops.c:1385:class_connect()) connect: client ea115449-6775-4569-9785-705ca0d938bc, cookie 0x58a604dcef35275e 00000100:00080000:2.0:1788572527.371056:0:110192:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572527.371069:0:110192:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc49237800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572527.371105:0:110192:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449863887488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572527.371128:0:110192:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import ea115449-6775-4569-9785-705ca0d938bc->MGS 00000080:00200000:2.0:1788572527.371146:0:110192:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc67142000 (sb ffff92fc67142000) 00000080:01000000:2.0:1788572527.371198:0:110192:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock,test_dummy_encryption 00000080:00000004:2.0:1788572527.371200:0:110192:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:2.0:1788572527.371204:0:110192:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:00000004:2.0:1788572527.371206:0:110192:0:(llite_lib.c:1242:ll_options()) next opt=test_dummy_encryption 00000080:01000000:2.0:1788572527.371214:0:110192:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: b5986402-2e9a-4d94-8ed3-664ccfdc4b41 10000000:01000000:2.0:1788572527.371622:0:110192:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:2.0:1788572527.371625:0:110192:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc67142000 10000000:01000000:2.0:1788572527.371629:0:110192:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc49933380 10000000:01000000:2.0:1788572527.371632:0:110192:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572527.371639:0:110192:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc49933380 from 1 10000000:01000000:2.0:1788572527.371642:0:110192:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788572527.371669:0:110192:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.371680:0:110192:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.371690:0:110192:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572527.371712:0:110192:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc4617ce00 x1875449863887616/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572527.371758:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322b40 going to sleep for 0 seconds 00000100:00100000:2.0:1788572527.371780:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ea115449-6775-4569-9785-705ca0d938bc:2392:1875449863887488:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572527.377108:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572527.377126:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:2.0:1788572527.377135:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:2.0:1788572527.377141:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:2.0:1788572527.377145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc49237800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:2.0:1788572527.377148:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572527.377203:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:2.0:1788572527.377216:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000007606db5f): req@ffff92fc4617ce00 x1875449863887616/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572527.377237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ea115449-6775-4569-9785-705ca0d938bc:2392:1875449863887488:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:2.0:1788572527.377365:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.384899:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863887616:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572527.384931:0:110192:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.384944:0:110192:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572527.385133:0:110192:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863887744:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572527.385245:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.396652:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863887744:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:2.0:1788572527.396700:0:110192:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572527.396713:0:110192:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.396746:0:110192:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572527.397105:0:110192:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc67142000 10000000:01000000:2.0:1788572527.397111:0:110192:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:2.0:1788572527.397117:0:110192:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc67142000 10000000:01000000:2.0:1788572527.397120:0:110192:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:2.0:1788572527.397123:0:110192:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc67142000 10000000:01000000:2.0:1788572527.397125:0:110192:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:2.0:1788572527.397129:0:110192:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc67142000 from 1 10000000:01000000:2.0:1788572527.397132:0:110192:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788572527.397178:0:110192:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.397187:0:110192:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.397199:0:110192:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572527.397224:0:110192:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863887872:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572527.397312:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.406341:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863887872:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572527.407311:0:110192:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.407327:0:110192:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572527.407417:0:110192:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863888000:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572527.407506:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.418342:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863888000:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572527.418416:0:110192:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863888128:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572527.418560:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.421678:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863888128:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572527.422361:0:110211:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110211:1875449863888256:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572527.422453:0:110211:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.425384:0:110211:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110211:1875449863888256:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572527.425432:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.425444:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.425446:0:110211:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1788572527.425454:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc67142000 00000020:00000080:2.0:1788572527.425457:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572527.425469:0:110211:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc67142000 (ffff92fc48cc99c0) 00000020:00000080:2.0:1788572527.425487:0:110211:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:2.0:1788572527.425492:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc67142000 00000020:00000080:2.0:1788572527.425494:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572527.425811:0:110211:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572527.425815:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.425819:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.425820:0:110211:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1788572527.425823:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.425826:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.425827:0:110211:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572527.425831:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc67142000 00000020:00000080:2.0:1788572527.425833:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572527.425838:0:110211:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc67142000 (ffff92fc48cc8000) 00000020:00000080:2.0:1788572527.425846:0:110211:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:2.0:1788572527.425850:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc67142000 00000020:00000080:2.0:1788572527.425852:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572527.427320:0:110211:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572527.427331:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.427337:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.427339:0:110211:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572527.427341:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.427345:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.427347:0:110211:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1788572527.427350:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1788572527.427352:0:110211:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1788572527.427354:0:110211:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1788572527.427360:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.427362:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.427364:0:110211:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1788572527.427366:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.427371:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.427372:0:110211:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1788572527.427375:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572527.427380:0:110211:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572527.427403:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc67142000 00000020:00000080:2.0:1788572527.427406:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572527.427416:0:110211:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc67142000 (ffff92fc48cccd40) 00000020:00000080:2.0:1788572527.427428:0:110211:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:2.0:1788572527.427433:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc67142000 00000020:00000080:2.0:1788572527.427435:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572527.427494:0:110211:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc69d43000@lustre-MDT0000-mdc-ffff92fc67142000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572527.433646:0:110211:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc67142000-1 is initialized 00000040:01000000:2.0:1788572527.434836:0:110211:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc67142000 ctxt 13 is initialized 00000020:00000080:2.0:1788572527.442655:0:110211:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572527.442673:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc67142000 00000020:00000080:2.0:1788572527.442677:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572527.442703:0:110211:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788572527.442767:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.442771:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.442773:0:110211:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788572527.442775:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572527.442779:0:110211:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:2.0:1788572527.442781:0:110211:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572527.442782:0:110211:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572527.442784:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572527.442786:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.442787:0:110211:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572527.442790:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.442793:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.442795:0:110211:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:2.0:1788572527.442797:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572527.442803:0:110211:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572527.442828:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc67142000 00000020:00000080:2.0:1788572527.442830:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572527.442841:0:110211:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc67142000 (ffff92fc48ccb380) 00000020:00000080:2.0:1788572527.442858:0:110211:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:2.0:1788572527.442862:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc67142000 00000020:00000080:2.0:1788572527.442864:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572527.443117:0:110211:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc69d45000@lustre-MDT0001-mdc-ffff92fc67142000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572527.448684:0:110211:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc67142000-2 is initialized 00000040:01000000:2.0:1788572527.449008:0:110211:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc67142000 ctxt 13 is initialized 00000020:00000080:2.0:1788572527.449213:0:110211:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572527.449227:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc67142000 00000020:00000080:2.0:1788572527.449232:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572527.449269:0:110211:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:2.0:1788572527.449301:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.449305:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.449306:0:110211:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:2.0:1788572527.449309:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572527.449312:0:110211:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:2.0:1788572527.449314:0:110211:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572527.449315:0:110211:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572527.449317:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572527.449389:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.449391:0:110211:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572527.449395:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572527.449399:0:110211:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572527.449403:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.449405:0:110211:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788572527.449408:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572527.449413:0:110211:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572527.449440:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc67142000 00000020:00000080:2.0:1788572527.449443:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572527.449454:0:110211:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc67142000 (ffff92fc509e4d40) 00000020:00000080:2.0:1788572527.449467:0:110211:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788572527.449472:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc67142000 00000020:00000080:2.0:1788572527.449475:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572527.449693:0:110211:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc69d46800@lustre-OST0000-osc-ffff92fc67142000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572527.449989:0:110211:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc67142000-3 is initialized 00000020:00000080:2.0:1788572527.450295:0:110211:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572527.450303:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc67142000 00000020:00000080:2.0:1788572527.450306:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572527.450387:0:110211:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788572527.450395:0:110211:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc67142000: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788572527.450400:0:110211:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788572527.450413:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.450416:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.450418:0:110211:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788572527.450421:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572527.450423:0:110211:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:2.0:1788572527.450424:0:110211:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572527.450426:0:110211:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572527.450427:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572527.450429:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.450431:0:110211:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572527.450433:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572527.450436:0:110211:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572527.450439:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.450440:0:110211:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788572527.450443:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572527.450447:0:110211:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572527.450461:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc67142000 00000020:00000080:2.0:1788572527.450464:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572527.450471:0:110211:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc67142000 (ffff92fc509e0000) 00000020:00000080:2.0:1788572527.450481:0:110211:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:2.0:1788572527.450486:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc67142000 00000020:00000080:2.0:1788572527.450488:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572527.450539:0:110211:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc69d40000@lustre-OST0001-osc-ffff92fc67142000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572527.450761:0:110211:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc67142000-4 is initialized 00000020:00000080:2.0:1788572527.451207:0:110211:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572527.451215:0:110211:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc67142000 00000020:00000080:2.0:1788572527.451219:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572527.451229:0:110211:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1788572527.451234:0:110211:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc67142000: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1788572527.451237:0:110211:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1788572527.451244:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.451248:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.451249:0:110211:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1788572527.451252:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.451255:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.451271:0:110211:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:2.0:1788572527.451275:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1788572527.451277:0:110211:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1788572527.451279:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.451282:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.451283:0:110211:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:2.0:1788572527.451404:0:110211:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110211:1875449863888384:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572527.451506:0:110211:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.463447:0:110211:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110211:1875449863888384:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572527.463499:0:110211:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.463511:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.463513:0:110211:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572527.463520:0:110211:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572527.463568:0:110211:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110211:1875449863888512:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572527.463664:0:110211:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572527.470471:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863888640:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572527.470510:0:110192:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.470524:0:110192:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572527.471136:0:110192:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ea115449-6775-4569-9785-705ca0d938bc:110192:1875449863888768:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:1.0:1788572527.471217:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705d8f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572527.499579:0:110216:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110216:1875449863889280:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572527.499681:0:110216:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705d8240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.508123:0:110216:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110216:1875449863889280:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572527.508185:0:110216:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.508197:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.508199:0:110216:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572527.508207:0:110216:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc67142000 00000020:00000080:2.0:1788572527.508211:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572527.508218:0:110216:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.508221:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.508222:0:110216:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572527.508224:0:110216:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.508226:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.508227:0:110216:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572527.508231:0:110216:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc67142000 00000020:00000080:2.0:1788572527.508233:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572527.508290:0:110216:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.508293:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.508294:0:110216:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:2.0:1788572527.508296:0:110216:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572527.508298:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.508299:0:110216:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:2.0:1788572527.508302:0:110216:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc67142000 00000020:00000080:2.0:1788572527.508304:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:2.0:1788572527.512622:0:110216:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 4310us 00000100:00100000:2.0:1788572527.512687:0:110216:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110216:1875449863889408:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572527.514119:0:110216:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc583226c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.528054:0:110216:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ea115449-6775-4569-9785-705ca0d938bc:110216:1875449863889408:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572527.528105:0:110216:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572527.528117:0:110216:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572527.528120:0:110216:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:2.0:1788572527.528125:0:110216:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:1.0:1788572527.528403:0:110192:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:1.0:1788572527.528422:0:110192:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788572527.528440:0:110192:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.528451:0:110192:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788572527.528470:0:110192:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1788572527.528494:0:110192:0:(genops.c:1385:class_connect()) connect: client b5986402-2e9a-4d94-8ed3-664ccfdc4b41, cookie 0x58a604dcef3527ab 00800000:01000000:1.0:1788572527.528515:0:110192:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect b5986402-2e9a-4d94-8ed3-664ccfdc4b41 to lustre-clilmv-ffff92fc67142000 00800000:01000000:1.0:1788572527.528523:0:110192:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc67142000(b5986402-2e9a-4d94-8ed3-664ccfdc4b41) - lustre-MDT0000_UUID, b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:00000080:1.0:1788572527.528530:0:110192:0:(genops.c:1385:class_connect()) connect: client b5986402-2e9a-4d94-8ed3-664ccfdc4b41, cookie 0x58a604dcef3527b2 00000100:00080000:1.0:1788572527.528538:0:110192:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67142000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572527.528548:0:110192:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67142000: import ffff92fc69d43000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572527.528561:0:110192:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572527.528584:0:110192:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449863889536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572527.528610:0:110192:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import b5986402-2e9a-4d94-8ed3-664ccfdc4b41->lustre-MDT0000_UUID 00800000:01000000:1.0:1788572527.528692:0:110192:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc67142000(b5986402-2e9a-4d94-8ed3-664ccfdc4b41) successfully (5) 00800000:00000004:1.0:1788572527.528696:0:110192:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc67142000: age 3283, max_age 0 00800000:01000000:1.0:1788572527.528718:0:110192:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc67142000(b5986402-2e9a-4d94-8ed3-664ccfdc4b41) - lustre-MDT0001_UUID, b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:00000080:1.0:1788572527.528748:0:110192:0:(genops.c:1385:class_connect()) connect: client b5986402-2e9a-4d94-8ed3-664ccfdc4b41, cookie 0x58a604dcef3527b9 00000100:00080000:1.0:1788572527.528762:0:110192:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67142000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572527.528766:0:110192:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67142000: import ffff92fc69d45000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572527.528772:0:110192:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572527.528781:0:110192:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449863889792/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572527.528790:0:110192:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import b5986402-2e9a-4d94-8ed3-664ccfdc4b41->lustre-MDT0001_UUID 00800000:01000000:1.0:1788572527.528827:0:110192:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc67142000(b5986402-2e9a-4d94-8ed3-664ccfdc4b41) successfully (5) 00800000:00000004:1.0:1788572527.528829:0:110192:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc67142000: age 3283, max_age 0 00000080:00000004:1.0:1788572527.528843:0:110192:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc67142000: age 3283, max_age 4282 00800000:00000004:1.0:1788572527.528847:0:110192:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc67142000: age 3283, max_age 4282 00000100:00080000:1.0:1788572527.528862:0:110192:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67289c00 x1875449863890048/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572527.528872:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705d8180 going to sleep for 0 seconds 00000100:00100000:2.0:1788572527.529002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863889536:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572527.529099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863889792:192.168.202.132@tcp:38:mount.lustre.0 00000100:00080000:0.0:1788572527.529652:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6728b800 x1875449863889664/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572527.529665:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:0.0:1788572527.529671:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67288a80 x1875449863889920/t0(0) o41->lustre-MDT0001-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572527.532790:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc67142000: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572527.532808:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc67142000: connect to target with instance 1 00000100:00080000:2.0:1788572527.532818:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc67142000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572527.532833:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1788572527.532836:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc69d43000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788572527.534588:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000e953d5d0): req@ffff92fc67289c00 x1875449863890048/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572527.534623:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc6728b800 x1875449863889664/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572527.534636:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863889536:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572527.534665:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc67142000: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572527.534675:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc67142000: connect to target with instance 1 00000100:00080000:2.0:1788572527.534683:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc67142000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572527.534692:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:2.0:1788572527.534695:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc69d45000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:0.0:1788572527.535565:0:110192:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705d8180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572527.535991:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863889792:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572527.548580:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:0:1875449863889920:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572527.548602:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:0:1875449863889664:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572527.551289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863890304:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572527.551350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863890432:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572527.564421:0:110192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110192:1875449863890688:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:2.0:1788572527.564588:0:110192:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00200000:2.0:1788572527.564596:0:110192:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:2.0:1788572527.564652:0:110192:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:2.0:1788572527.564656:0:110192:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00010000:00010000:1.0:1788572527.789621:0:110237:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:1.0:1788572527.789646:0:110237:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc67142000 00010000:00010000:1.0:1788572527.789658:0:110237:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc67142000 00010000:00010000:1.0:1788572527.789669:0:110237:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc67142000 00010000:00010000:1.0:1788572527.789680:0:110237:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc67142000 00000080:00200000:1.0:1788572527.847507:0:110241:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88),name=/ 00000002:00010000:1.0:1788572527.847543:0:110241:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572527.847608:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.847620:0:110241:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.847630:0:110241:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110241 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572527.847657:0:110241:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job cmp:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110241:1875449863890816:192.168.202.132@tcp:101:cmp.0 00000100:00100000:1.0:1788572527.847773:0:110241:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705d8a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.854555:0:110241:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job cmp:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110241:1875449863890816:192.168.202.132@tcp:101:cmp.0 00010000:00010000:2.0:1788572527.854587:0:110241:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854602:0:110241:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854620:0:110241:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854636:0:110241:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854650:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854659:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854667:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572527.854680:0:110241:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b800 x1875449863890816/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572543 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572527.854695:0:110241:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854709:0:110241:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc583c5000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572527.854716:0:110241:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572527.854797:0:110241:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572527.854803:0:110241:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef3527d5 00000080:00010000:2.0:1788572527.854809:0:110241:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572527.854813:0:110241:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3527d5 from it ffff9e0384223bf8 00010000:00010000:2.0:1788572527.854817:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854837:0:110241:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572527.854867:0:110241:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572527.854891:0:110241:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e3a88), flags=25 00000080:00200000:2.0:1788572527.854896:0:110241:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e3a88), intent=0 00000002:00010000:2.0:1788572527.854923:0:110241:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572527.854968:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854977:0:110241:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.854986:0:110241:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110241 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572527.855024:0:110241:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job cmp:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110241:1875449863890944:192.168.202.132@tcp:101:cmp.0 00000100:00100000:2.0:1788572527.855106:0:110241:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc583226c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572527.863601:0:110241:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job cmp:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110241:1875449863890944:192.168.202.132@tcp:101:cmp.0 00010000:00010000:2.0:1788572527.863633:0:110241:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.863658:0:110241:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.863666:0:110241:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.863679:0:110241:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.863690:0:110241:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.863704:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.863712:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.866420:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572527.866453:0:110241:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617fb80 x1875449863890944/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572543 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'cmp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572527.866469:0:110241:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.866484:0:110241:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc583c5000 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:2.0:1788572527.866490:0:110241:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572527.866550:0:110241:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e4b08) 00000080:00200000:2.0:1788572527.866558:0:110241:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572527.866575:0:110241:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3527dc 00000080:00010000:2.0:1788572527.866588:0:110241:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e4b08) 00000080:00010000:2.0:1788572527.866592:0:110241:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3527dc from it ffff9e0384223ac8 00010000:00010000:2.0:1788572527.866596:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.866606:0:110241:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366e00/0x58a604dcef3527dc lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929fb expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572527.866647:0:110241:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e4b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572527.866669:0:110241:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f59b.sanity-sec, dir=[0x20000041d:0x1:0x0](ffff92fc665e4b08), file ffff92fc43b00480, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1788572527.866677:0:110241:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f59b.sanity-sec, dir=[0x20000041d:0x1:0x0](ffff92fc665e4b08), intent=open 00000080:00200000:0.0:1788572528.038691:0:110241:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788572528.038709:0:110241:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:0.0:1788572528.039216:0:110241:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572528.039221:0:110241:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f2eb0, osc lock ffff92fc45932ab0, flags 0x400001002 00010000:00010000:0.0:1788572528.039415:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 3/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.039431:0:110241:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 3/1,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572528.039912:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2394:1875449863891456:192.168.202.132@tcp:101:cmp.0 00000020:00200000:0.0:1788572528.040046:0:110241:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572528.045191:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.045222:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.045239:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.045267:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572528.045283:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 9000, setting rss=9000; kms=9000, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572528.045295:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788572528.045318:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.045328:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.045420:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572528.045431:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572528.045518:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2394:1875449863891456:192.168.202.132@tcp:101:cmp.0 00000080:00200000:1.0:1788572528.045634:0:110241:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:1.0:1788572528.045640:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788572528.045644:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.045649:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f2eb0 (ffff92fc582556c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.045652:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572528.045655:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f2eb0 (ffff92fc58255a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572528.045667:0:110241:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/1, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788572528.046247:0:110241:0:(file.c:5214:ll_file_seek()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_CUR 00000080:00200008:1.0:1788572528.046267:0:110241:0:(file.c:5245:ll_file_seek()) COMPLETED file f59b.sanity-sec:[0x20000041d:0x2:0x0], offset: 0, type: SEEK_CUR, rc = 0 00000080:00200008:1.0:1788572528.046333:0:110241:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 0, count: 4194304 00000080:00200000:1.0:1788572528.046363:0:110241:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:1.0:1788572528.046386:0:110241:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 0, bytes: 4194304 00000080:00200000:1.0:1788572528.046391:0:110241:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 0 00000020:00200000:1.0:1788572528.046398:0:110241:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d945bd9d[0x110, 1, [0x20000041d:0x2:0x0] hash] 00000020:00200000:1.0:1788572528.046400:0:110241:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:1.0:1788572528.046404:0:110241:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 3 restore needed 0 00020000:00200000:1.0:1788572528.046419:0:110241:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:1.0:1788572528.046423:0:110241:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572528.046438:0:110241:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:1.0:1788572528.046579:0:110241:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572528.046585:0:110241:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d47b0, osc lock ffff92fc47ff5430, flags 0x0 00010000:00010000:1.0:1788572528.046606:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046620:0:110241:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046632:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046640:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046648:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572528.046657:0:110241:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [0, 4194304) 00000001:00010000:1.0:1788572528.046670:0:110241:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:1.0:1788572528.046675:0:110241:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572528.046677:0:110241:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d43b0, osc lock ffff92fc47ff51d0, flags 0x400001002 00010000:00010000:1.0:1788572528.046684:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046694:0:110241:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788572528.046702:0:110241:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572528.046706:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046714:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788572528.046830:0:110241:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572528.046835:0:110241:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:1.0:1788572528.046840:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788572528.046843:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.046845:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d43b0 (ffff92fc582556c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.046848:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572528.046850:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d43b0 (ffff92fc58255a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572528.046858:0:110241:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00010000:00010000:1.0:1788572528.046895:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046906:0:110241:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (4096 8191) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046953:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/3,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.046962:0:110241:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (12288 16383) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/3,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788572528.046987:0:110241:0:(osc_cache.c:3059:osc_queue_sync_pages()) ### no dlm lock: extent: ffff92fc451f5780 00010000:00010000:1.0:1788572528.047005:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 8/3,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.047013:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.047022:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.047164:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788572528.047221:0:103957:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc451f5780 00010000:00010000:1.0:1788572528.047242:0:103957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:1.0:1788572528.047294:0:103957:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc67288a80 - object 0x280000401:552 offset 0<>12288 00000100:00100000:0.0:1788572528.047440:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2393:1875449863891584:192.168.202.132@tcp:3:cmp.0 00000008:00010000:0.0:1788572528.057704:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc451f5780 00000020:00200000:0.0:1788572528.057824:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1788572528.057837:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc451f5780 00000100:00100000:0.0:1788572528.057863:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2393:1875449863891584:192.168.202.132@tcp:3:cmp.0 00000020:00010000:1.0:1788572528.059527:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804cf38 (ffff92fc58255a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788572528.059536:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804cf38 (ffff92fc58255a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.059541:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d47b0 (ffff92fc582556c8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572528.059552:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572528.059569:0:110241:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00010000:1.0:1788572528.059591:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804cf38 (ffff92fc58255a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572528.059595:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d47b0 (ffff92fc58255a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572528.059605:0:110241:0:(vvp_io.c:533:vvp_io_advance()) advancing 9000 bytes 00000080:00200000:1.0:1788572528.059620:0:110241:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572528.059625:0:110241:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 9000, restart: 0 00000080:00200000:1.0:1788572528.059630:0:110241:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 9000 00000080:00200008:1.0:1788572528.059646:0:110241:0:(file.c:2483:do_file_read_iter()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0], ppos: 9000, count: 4185304 00000080:00200000:1.0:1788572528.059656:0:110241:0:(file.c:2090:ll_file_io_generic()) f59b.sanity-sec: read ppos: 9000, bytes: 4185304 00000080:00200000:1.0:1788572528.059661:0:110241:0:(file.c:1848:ll_io_set_mirror()) f59b.sanity-sec: desiginated mirror: 0 00000020:00200000:1.0:1788572528.059667:0:110241:0:(cl_io.c:214:cl_io_rw_init()) header@00000000d945bd9d[0x110, 4, [0x20000041d:0x2:0x0] hash] 00000020:00200000:1.0:1788572528.059669:0:110241:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [9000, 4194304) 0 0 00000080:00200000:1.0:1788572528.059673:0:110241:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 3 restore needed 0 00020000:00200000:1.0:1788572528.059686:0:110241:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 9000 chunk: [9000, 4194304) 4194304, 4185304 00020000:00200000:1.0:1788572528.059690:0:110241:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572528.059707:0:110241:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [2, 1023] 00000008:00010000:1.0:1788572528.059745:0:110241:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572528.060520:0:110241:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d47b0, osc lock ffff92fc47ff5430, flags 0x0 00010000:00010000:1.0:1788572528.060551:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.060568:0:110241:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (8192 4194303) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.060583:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.060591:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.060599:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572528.060609:0:110241:0:(vvp_io.c:835:vvp_io_read_start()) f59b.sanity-sec: read [9000, 4194304) 00000001:00010000:1.0:1788572528.060625:0:110241:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20000041d:0x2:0x0] 00000008:00010000:1.0:1788572528.060635:0:110241:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572528.060637:0:110241:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d43b0, osc lock ffff92fc47ff51d0, flags 0x400001002 00010000:00010000:1.0:1788572528.060644:0:110241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.060654:0:110241:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572528.060664:0:110241:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572528.060667:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.060674:0:110241:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572528.060681:0:110241:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572528.060687:0:110241:0:(file.c:1775:ll_merge_attr_nolock()) [0x20000041d:0x2:0x0] updating i_size 9000 i_blocks 24 00000020:00010000:1.0:1788572528.060691:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788572528.060693:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.060696:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d43b0 (ffff92fc582556c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.060699:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49fbd970 (ffff92fc58255a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572528.060701:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d43b0 (ffff92fc58255a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572528.060709:0:110241:0:(vvp_io.c:893:vvp_io_read_start()) count:4185304 iocb pos:9000 00000020:00010000:1.0:1788572528.060717:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc4804cf38 (ffff92fc58255a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788572528.061328:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc4804cf38 (ffff92fc58255a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572528.061331:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d47b0 (ffff92fc582556c8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572528.061350:0:110241:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572528.061363:0:110241:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:1.0:1788572528.061377:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc4804cf38 (ffff92fc58255a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572528.061379:0:110241:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d47b0 (ffff92fc58255a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572528.061387:0:110241:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:1.0:1788572528.061397:0:110241:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 0/0, layout version 3 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572528.061400:0:110241:0:(file.c:2238:ll_file_io_generic()) f59b.sanity-sec: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:1.0:1788572528.061403:0:110241:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:1.0:1788572528.061475:0:110241:0:(file.c:479:ll_file_release()) START file f59b.sanity-sec:[0x20000041d:0x2:0x0](ffff92fc665d9148), flags 100000 00000002:00100000:1.0:1788572528.061549:0:110241:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67339500 x1875449863891328/t4294983740(4294983740) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 672/776 e 0 to 0 dl 1788572544 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cmp.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572528.061560:0:110241:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc67142000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572528.061565:0:110241:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67289180 x1875449863891840/t0(0) o35->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572528.061587:0:110241:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job cmp:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110241:1875449863891840:192.168.202.132@tcp:35:cmp.0 00000100:00100000:1.0:1788572528.061677:0:110241:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc705d8780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572528.072151:0:110241:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job cmp:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110241:1875449863891840:192.168.202.132@tcp:35:cmp.0 00000002:00100000:0.0:1788572528.072188:0:110241:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67339500 x1875449863891328/t4294983740(4294983740) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 672/776 e 0 to 0 dl 1788572544 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cmp.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572528.394538:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572529.412984:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572530.436901:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572531.460956:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572532.490088:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572533.509092:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572534.541075:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572534.641519:0:110671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863893760:192.168.202.132@tcp:36:rm.0 00000080:00200000:1.0:1788572534.641580:0:110671:0:(vvp_io.c:1826:vvp_io_init()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 3 restore needed 0 00020000:00200000:1.0:1788572534.641593:0:110671:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572534.641622:0:110671:0:(vvp_io.c:304:vvp_io_fini()) [0x20000041d:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572534.641712:0:110671:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x2:0x0](ffff92fc665d9148) 00000008:00010000:1.0:1788572534.641859:0:110671:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 9000 blocks 24 [cma]time: 1788572511 1788572511 1788572496 by oinfo size 9000 blocks 24 [cma]time 1788572511 1788572511 1788572528 ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 1/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200008:1.0:1788572534.641965:0:110671:0:(file.c:479:ll_file_release()) START file d59b.sanity-sec:[0x20000041d:0x1:0x0](ffff92fc665e4b08), flags 704000 00010000:00010000:1.0:1788572534.641995:0:110671:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70204400/0x58a604dcef3527f8 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a48 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572534.642043:0:110671:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88),name=/ 00000002:00010000:1.0:1788572534.642061:0:110671:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572534.642073:0:110671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572534.642083:0:110671:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788572534.642090:0:110671:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572534.642096:0:110671:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:1.0:1788572534.642100:0:110671:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef3527d5 00000080:00010000:1.0:1788572534.642108:0:110671:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3527d5 from it ffff9e03816abc10 00010000:00010000:1.0:1788572534.642112:0:110671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572534.642120:0:110671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572534.642138:0:110671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572534.642163:0:110671:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e3a88), flags=0 00000080:00200000:1.0:1788572534.642168:0:110671:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e3a88), intent=getattr 00000002:00010000:1.0:1788572534.642183:0:110671:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59b.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572534.642222:0:110671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110671 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.642240:0:110671:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110671 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.642250:0:110671:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110671 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572534.642272:0:110671:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863893888:192.168.202.132@tcp:101:rm.0 00000100:00100000:1.0:1788572534.642510:0:110671:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572534.654264:0:110671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863893888:192.168.202.132@tcp:101:rm.0 00010000:00010000:1.0:1788572534.654290:0:110671:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654308:0:110671:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654316:0:110671:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 4/1,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654337:0:110671:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654349:0:110671:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654364:0:110671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654372:0:110671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 5/2,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654380:0:110671:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 4/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572534.654392:0:110671:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67288700 x1875449863893888/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572550 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572534.654405:0:110671:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654416:0:110671:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc583c5000 type 13 mode 4 res 8589935645/1 (0 0) 00000080:00200000:1.0:1788572534.654422:0:110671:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572534.654448:0:110671:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20000041d:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572534.654451:0:110671:0:(namei.c:147:ll_iget()) got inode: [0x20000041d:0x1:0x0](ffff92fc665e4b08): rc = 0 00000080:00010000:1.0:1788572534.654463:0:110671:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e4b08) for lock 0x58a604dcef3527ff 00000080:00010000:1.0:1788572534.654474:0:110671:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20000041d:0x1:0x0](ffff92fc665e4b08) 00000080:00010000:1.0:1788572534.654478:0:110671:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3527ff from it ffff9e03816abe00 00010000:00010000:1.0:1788572534.654481:0:110671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 3/1,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572534.654489:0:110671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572534.654523:0:110671:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88),name=/ 00000002:00010000:1.0:1788572534.654541:0:110671:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572534.654550:0:110671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788572534.654560:0:110671:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572534.654567:0:110671:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572534.654571:0:110671:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:1.0:1788572534.654574:0:110671:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef3527d5 00000080:00010000:1.0:1788572534.654577:0:110671:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3527d5 from it ffff9e03816abd88 00010000:00010000:1.0:1788572534.654579:0:110671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1788572534.654586:0:110671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788572534.654595:0:110671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572534.654601:0:110671:0:(namei.c:2261:ll_rmdir()) VFS Op:name=d59b.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e3a88) 00010000:00010000:1.0:1788572534.654621:0:110671:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572534.654643:0:110671:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 2/0,0 mode: PR/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572534.654654:0:110671:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70204400/0x58a604dcef3527f8 lrc: 2/0,0 mode: CR/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792a48 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:1.0:1788572534.654695:0:110671:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6733b800 x1875449863893376/t4294983744(4294983744) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572550 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572534.654711:0:110671:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc67142000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572534.654717:0:110671:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67288700 x1875449863894144/t0(0) o35->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572534.654780:0:110671:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863894144:192.168.202.132@tcp:35:rm.0 00000100:00100000:1.0:1788572534.655158:0:110671:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572534.659860:0:110671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863894144:192.168.202.132@tcp:35:rm.0 00000002:00100000:1.0:1788572534.659898:0:110671:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6733b800 x1875449863893376/t4294983744(4294983744) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572550 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572534.659977:0:110671:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 1/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572534.659988:0:110671:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70204400/0x58a604dcef3527f8 lrc: 1/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792a48 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572534.659995:0:110671:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572534.660001:0:110671:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788572534.660004:0:110671:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f5e00/0x58a604dcef3527ff lrc: 0/0,0 mode: --/PR res: [0x20000041d:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792a87 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572534.660016:0:110671:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70204400/0x58a604dcef3527f8 lrc: 0/0,0 mode: --/CR res: [0x20000041d:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792a48 expref: -99 pid: 110671 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572534.660029:0:110671:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc702f4a00/0x58a604dcef3527d5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217929df expref: -99 pid: 110241 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:1.0:1788572534.660053:0:110671:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc67142000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572534.660060:0:110671:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67288700 x1875449863894400/t0(0) o36->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 504/736 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572534.660082:0:110671:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863894400:192.168.202.132@tcp:36:rm.0 00000100:00100000:1.0:1788572534.660166:0:110671:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572534.666253:0:110671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job rm:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110671:1875449863894400:192.168.202.132@tcp:36:rm.0 00000080:00200000:1.0:1788572534.666300:0:110671:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000041d:0x1:0x0](ffff92fc665e4b08) 00000080:00200008:2.0:1788572535.112864:0:110687:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572535.112882:0:110687:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88),name=/ 00000002:00010000:2.0:1788572535.112909:0:110687:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572535.112973:0:110687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.112984:0:110687:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.112993:0:110687:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110687 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572535.113017:0:110687:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job lsof:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110687:1875449863894528:192.168.202.132@tcp:101:lsof.0 00000100:00100000:2.0:1788572535.113252:0:110687:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4384e240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572535.116496:0:110687:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job lsof:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110687:1875449863894528:192.168.202.132@tcp:101:lsof.0 00010000:00010000:2.0:1788572535.116522:0:110687:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116541:0:110687:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116554:0:110687:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116564:0:110687:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116577:0:110687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116584:0:110687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116592:0:110687:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572535.116604:0:110687:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617fb80 x1875449863894528/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572551 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572535.116618:0:110687:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116632:0:110687:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc583c5000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572535.116639:0:110687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572535.116669:0:110687:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572535.116676:0:110687:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352806 00000080:00010000:2.0:1788572535.116682:0:110687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572535.116685:0:110687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352806 from it ffff9e0381c1fcb8 00010000:00010000:2.0:1788572535.116689:0:110687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572535.116697:0:110687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572535.567504:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572536.587557:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572537.610190:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572537.861862:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863894784:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.861986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863895040:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.865898:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788572537.865930:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:ea115449-6775-4569-9785-705ca0d938bc:2396:1875449863894656:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.865963:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863894912:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.865987:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863895168:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.868227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863894784:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.869384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:ea115449-6775-4569-9785-705ca0d938bc:2396:1875449863894656:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.869403:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863894912:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.869413:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863895040:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572537.873433:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2396:1875449863895168:192.168.202.132@tcp:400:kworker.0 00000080:00200008:1.0:1788572538.463163:0:110695:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572538.463176:0:110695:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88),name=/ 00000002:00010000:1.0:1788572538.463197:0:110695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572538.463223:0:110695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572538.463240:0:110695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572538.463246:0:110695:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572538.463253:0:110695:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:1.0:1788572538.463256:0:110695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352806 00000080:00010000:1.0:1788572538.463262:0:110695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352806 from it ffff9e0380da3cb8 00010000:00010000:1.0:1788572538.463265:0:110695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572538.463276:0:110695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:1.0:1788572538.463394:0:110695:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572538.463398:0:110695:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88),name=/ 00000002:00010000:1.0:1788572538.463404:0:110695:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572538.463409:0:110695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572538.463414:0:110695:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572538.463419:0:110695:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572538.463422:0:110695:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:1.0:1788572538.463424:0:110695:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352806 00000080:00010000:1.0:1788572538.463426:0:110695:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352806 from it ffff9e0380da3cb8 00010000:00010000:1.0:1788572538.463429:0:110695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572538.463433:0:110695:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572538.484914:0:110695:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e3a88) 00000080:00200000:1.0:1788572538.484954:0:110695:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc67142000 (sb ffff92fc67142000) 00010000:00010000:1.0:1788572538.484997:0:110695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485015:0:110695:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485022:0:110695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485030:0:110695:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485101:0:110695:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572538.485108:0:110695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485113:0:110695:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485117:0:110695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485120:0:110695:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485142:0:110204:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788572538.485149:0:110204:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485154:0:110204:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367000 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572538.485157:0:110204:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485163:0:110204:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572538.485168:0:110204:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572538.485172:0:110204:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572538.485174:0:110204:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:2.0:1788572538.485207:0:110204:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485211:0:110204:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572538.485258:0:110204:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485266:0:110204:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70367000/0x58a604dcef35276c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929a7 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485279:0:110204:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572538.485282:0:110204:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485288:0:110204:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205c00 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572538.485291:0:110204:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485298:0:110204:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 4/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572538.485304:0:110204:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 4/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572538.485309:0:110204:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572538.485311:0:110204:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 00010000:00010000:2.0:1788572538.485323:0:110204:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 3/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485328:0:110204:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572538.485334:0:110204:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485341:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572538.485343:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572538.485345:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572538.485364:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:ea115449-6775-4569-9785-705ca0d938bc:2395:1875449863895424:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572538.485467:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572538.485473:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:ea115449-6775-4569-9785-705ca0d938bc:2395:1875449863895296:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572538.485602:0:110695:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:1.0:1788572538.485607:0:110695:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef35279d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929ae expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572538.485622:0:110695:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572538.485629:0:110695:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485636:0:110695:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485641:0:110695:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.485647:0:110695:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485675:0:110203:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572538.485680:0:110203:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485685:0:110203:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204200 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572538.485688:0:110203:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485700:0:110203:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572538.485706:0:110203:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 4/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572538.485711:0:110203:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:2.0:1788572538.485713:0:110203:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 00010000:00010000:2.0:1788572538.485827:0:110203:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 3/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485834:0:110203:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572538.485846:0:110203:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.485853:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572538.485854:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572538.485856:0:110203:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572538.485866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:ea115449-6775-4569-9785-705ca0d938bc:2396:1875449863895552:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572538.485962:0:110695:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:1.0:1788572538.485966:0:110695:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef3527a4 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929b5 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572538.485977:0:110695:0:(mgc_request.c:130:config_log_put()) dropping config log params 10000000:01000000:1.0:1788572538.485980:0:110695:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:1.0:1788572538.485982:0:110695:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:1.0:1788572538.486469:0:110695:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc51c47000 active: 1 00020000:01000000:1.0:1788572538.486473:0:110695:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc51c43000 active: 1 00020000:01000000:1.0:1788572538.486474:0:110695:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:1.0:1788572538.486478:0:110695:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc67142000 00020000:01000000:1.0:1788572538.486480:0:110695:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc67142000: disconnecting target lustre-OST0001-osc-ffff92fc67142000 00000100:00080000:1.0:1788572538.486490:0:110695:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import b5986402-2e9a-4d94-8ed3-664ccfdc4b41->lustre-OST0001_UUID 00000100:00080000:1.0:1788572538.486564:0:110695:0:(import.c:65:import_set_state_nolock()) ffff92fc69d40000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:2.0:1788572538.486621:0:110204:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572538.486632:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863895680:192.168.202.132@tcp:9:umount.0 00000100:00100000:2.0:1788572538.493129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:ea115449-6775-4569-9785-705ca0d938bc:2396:1875449863895552:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572538.493167:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:ea115449-6775-4569-9785-705ca0d938bc:2395:1875449863895296:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572538.493176:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:ea115449-6775-4569-9785-705ca0d938bc:2395:1875449863895424:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:1.0:1788572538.493436:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572538.493441:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc67142000: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572538.493446:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572538.493576:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3527ce 00000020:00000080:1.0:1788572538.493593:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc51c43000/LOV_OSC_UUID 00000020:01000000:1.0:1788572538.493644:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc67142000 (flags='') 00000020:00000080:1.0:1788572538.493650:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572538.493658:0:110695:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc67142000: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572538.493663:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc509e0000) has no exports 00000020:00000080:2.0:1788572538.494037:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51c43000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc67142000 00000008:01000000:1.0:1788572538.494198:0:110695:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc67142000: client import never connected 00000100:00080000:1.0:1788572538.494201:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572538.494203:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc67142000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572538.494207:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572538.494419:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572538.494426:0:110695:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00020000:01000000:1.0:1788572538.494430:0:110695:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc67142000 00020000:01000000:1.0:1788572538.494432:0:110695:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc67142000: disconnecting target lustre-OST0000-osc-ffff92fc67142000 00000100:00080000:1.0:1788572538.494442:0:110695:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import b5986402-2e9a-4d94-8ed3-664ccfdc4b41->lustre-OST0000_UUID 00010000:00010000:1.0:1788572538.494471:0:110695:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x228:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572538.494511:0:110695:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 1/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572538.494519:0:110695:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572538.494533:0:110695:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfd80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572538.494540:0:110695:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job umount:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110695:1875449863895808:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572538.494549:0:110695:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572538.494554:0:110695:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572538.494557:0:110695:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc67142000 lock: 00000000acb18b86/0x58a604dcef3527ea lrc: 0/0,0 mode: --/PR res: [0x280000401:0x228:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792a2c expref: -99 pid: 110241 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00080000:1.0:1788572538.494640:0:110695:0:(import.c:65:import_set_state_nolock()) ffff92fc69d46800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572538.499920:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572538.499924:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc67142000: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572538.499929:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572538.500210:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3527c7 00000020:00000080:1.0:1788572538.500334:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc51c47000/LOV_OSC_UUID 00000020:00000080:2.0:1788572538.501102:0:110631:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc69d40000 for lustre-OST0001-osc-ffff92fc67142000 00000020:00000080:2.0:1788572538.501113:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc69d45800/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:00000080:2.0:1788572538.501117:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d45800/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 for lustre-OST0001-osc-ffff92fc67142000 00000020:01000000:2.0:1788572538.501127:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc67142000 (b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:1.0:1788572538.506052:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc67142000 (flags='') 00000020:00000080:1.0:1788572538.506059:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572538.506067:0:110695:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc67142000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572538.506072:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc509e4d40) has no exports 00000008:01000000:1.0:1788572538.506352:0:110695:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc67142000: client import never connected 00000100:00080000:1.0:1788572538.506358:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572538.506362:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc67142000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572538.506369:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572538.506583:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572538.506589:0:110695:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:00000080:1.0:1788572538.506595:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3527c0 00000020:00000080:1.0:1788572538.506601:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc67142800/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000100:00080000:0.0:1788572538.521871:0:110695:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import b5986402-2e9a-4d94-8ed3-664ccfdc4b41->lustre-MDT0000_UUID 00010000:00010000:0.0:1788572538.521988:0:110695:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572538.522054:0:110695:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572538.522061:0:110695:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572538.522078:0:110695:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48e0e6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572538.522089:0:110695:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job umount:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:110695:1875449863896064:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572538.522098:0:110695:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572538.522105:0:110695:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572538.522109:0:110695:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67142000 lock: ffff92fc70366c00/0x58a604dcef352806 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792aaa expref: -99 pid: 110687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00080000:0.0:1788572538.523249:0:110695:0:(import.c:65:import_set_state_nolock()) ffff92fc69d43000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572538.523466:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863896192:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572538.528960:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc69d43000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572538.529001:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863896192:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572538.529088:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572538.529093:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc67142000: committing for last_committed 4294983741 gen 2 00000100:00100000:0.0:1788572538.529103:0:110695:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67338000 x1875449863893760/t4294983745(4294983745) o36->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 504/456 e 0 to 0 dl 1788572550 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.529124:0:110695:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67288700 x1875449863894400/t4294983747(4294983747) o36->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 504/456 e 0 to 0 dl 1788572550 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.529134:0:110695:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6704b100 x1875449863896192/t4294983748(4294983748) o39->lustre-MDT0000-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572548 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.529146:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572538.530920:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3527b2 00000020:00000080:0.0:1788572538.530933:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc51d2b800/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000100:00080000:0.0:1788572538.531034:0:110695:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import b5986402-2e9a-4d94-8ed3-664ccfdc4b41->lustre-MDT0001_UUID 00000100:00080000:0.0:1788572538.532041:0:110695:0:(import.c:65:import_set_state_nolock()) ffff92fc69d45000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572538.532204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863896320:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572538.538390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc69d45000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572538.538424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b5986402-2e9a-4d94-8ed3-664ccfdc4b41:2392:1875449863896320:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572538.538482:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572538.538487:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc67142000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572538.538495:0:110695:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6704b100 x1875449863896320/t4294976750(4294976750) o39->lustre-MDT0001-mdc-ffff92fc67142000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572548 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.538525:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572538.543363:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3527b9 00000020:00000080:0.0:1788572538.543379:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc51d2f000/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:00000080:2.0:1788572538.543563:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3527ab 00000020:00000080:2.0:1788572538.543570:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc51d2c800/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:01000000:2.0:1788572538.544144:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc67142000 (flags='') 00000020:00000080:2.0:1788572538.544155:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572538.544162:0:110695:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc67142000: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572538.544167:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc48cc99c0) has no exports 00000020:00000080:2.0:1788572538.544246:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572538.544261:0:110695:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:00000080:2.0:1788572538.544266:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc69d44000/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:00000080:2.0:1788572538.544268:0:110695:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d44000/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 for lustre-clilov-ffff92fc67142000 00000020:01000000:2.0:1788572538.544273:0:110695:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc67142000 (b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572538.544450:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc67142000 (flags='') 00000020:00000080:2.0:1788572538.544454:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572538.544458:0:110695:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc67142000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572538.544461:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc48cc8000) has no exports 00000020:00000080:2.0:1788572538.544502:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572538.544506:0:110695:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572538.544509:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc67142000 (flags='') 00000020:00000080:2.0:1788572538.544523:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572538.544527:0:110695:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc67142000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572538.544529:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc48cccd40) has no exports 00000008:01000000:2.0:1788572538.545850:0:110695:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc67142000: client import never connected 00000100:00080000:2.0:1788572538.545855:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572538.545859:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc67142000: committing for last_committed 4294983741 gen 3 00000100:00100000:2.0:1788572538.545867:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572538.550030:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572538.550042:0:110695:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000020:01000000:2.0:1788572538.550058:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc67142000 (flags='') 00000020:00000080:2.0:1788572538.550062:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572538.550066:0:110695:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc67142000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572538.550071:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc48ccb380) has no exports 00000008:01000000:1.0:1788572538.551663:0:110695:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc67142000: client import never connected 00000100:00080000:1.0:1788572538.551670:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572538.551674:0:110695:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc67142000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572538.551682:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572538.557165:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572538.557180:0:110695:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc67142000 (uuid b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000080:02000400:1.0:1788572538.557185:0:110695:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572538.557188:0:110695:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1788572538.557249:0:110695:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc67142000 00000100:00080000:1.0:1788572538.558623:0:110695:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import ea115449-6775-4569-9785-705ca0d938bc->MGS 00010000:00010000:1.0:1788572538.558640:0:110695:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572538.558653:0:110695:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572538.558659:0:110695:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572538.558661:0:110695:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572538.558664:0:110695:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572538.561275:0:110695:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572538.561293:0:110695:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572538.561314:0:110695:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfd80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572538.561324:0:110695:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job umount:ea115449-6775-4569-9785-705ca0d938bc:110695:1875449863896448:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572538.561336:0:110695:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572538.561345:0:110695:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572538.561350:0:110695:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366600/0x58a604dcef352765 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217929a0 expref: -99 pid: 110192 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572538.561392:0:110695:0:(import.c:65:import_set_state_nolock()) ffff92fc49237800 MGS: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572538.561428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ea115449-6775-4569-9785-705ca0d938bc:2392:1875449863896576:192.168.202.132@tcp:251:umount.0 00000020:00000080:2.0:1788572538.561742:0:110631:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc69d45000 for lustre-MDT0001-mdc-ffff92fc67142000 00000020:00000080:2.0:1788572538.561754:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc69d42000/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 00000020:00000080:2.0:1788572538.561758:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d42000/b5986402-2e9a-4d94-8ed3-664ccfdc4b41 for lustre-MDT0001-mdc-ffff92fc67142000 00000020:01000000:2.0:1788572538.561766:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc67142000 (b5986402-2e9a-4d94-8ed3-664ccfdc4b41) 00000100:00080000:1.0:1788572538.570827:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc49237800 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:1.0:1788572538.570840:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788572538.570843:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788572538.570844:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572538.570867:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ea115449-6775-4569-9785-705ca0d938bc:2392:1875449863896576:192.168.202.132@tcp:251:umount.0 00000100:00080000:0.0:1788572538.570929:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:0.0:1788572538.570934:0:110695:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572538.570937:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788572538.570939:0:110695:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:0.0:1788572538.570950:0:110695:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35275e 00000020:00000080:0.0:1788572538.570970:0:110695:0:(genops.c:885:class_export_put()) final put ffff92fc4990f800/ea115449-6775-4569-9785-705ca0d938bc 00000020:01000000:0.0:1788572538.570992:0:110695:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:0.0:1788572538.570999:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572538.571007:0:110695:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572538.571012:0:110695:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc49933380) has no exports 00000020:00000080:1.0:1788572538.571094:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990f800/ea115449-6775-4569-9785-705ca0d938bc for MGC192.168.202.132@tcp 10000000:01000000:2.0:1788572538.571493:0:110207:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:0.0:1788572538.571713:0:110695:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:0.0:1788572538.571716:0:110695:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:0.0:1788572538.571740:0:110695:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572538.571743:0:110695:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788572538.571745:0:110695:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:0.0:1788572538.571812:0:110695:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572538.571818:0:110695:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid ea115449-6775-4569-9785-705ca0d938bc) 00000020:01000004:0.0:1788572538.571823:0:110695:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc67142000 1 00000020:01000004:0.0:1788572538.571827:0:110695:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc67140800 00000020:00000080:1.0:1788572538.572002:0:110631:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc49237800 for MGC192.168.202.132@tcp 00000020:00000080:1.0:1788572538.572010:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc67144000/ea115449-6775-4569-9785-705ca0d938bc 00000020:00000080:1.0:1788572538.572012:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67144000/ea115449-6775-4569-9785-705ca0d938bc for MGC192.168.202.132@tcp 00000020:01000000:1.0:1788572538.572017:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (ea115449-6775-4569-9785-705ca0d938bc) 00010000:00010000:1.0:1788572538.572302:0:110203:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572538.572427:0:110204:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00000100:00100000:1.0:1788572538.572566:0:110194:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc4900ce00:110194] 1 exiting: rc = 0 00000100:00100000:0.0:1788572538.573371:0:110193:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc4900d200:110193] 0 exiting: rc = 1 00000100:00100000:2.0:1788572538.577594:0:110195:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc47f04600:110195] 0 exiting: rc = 1 00000100:00100000:0.0:1788572538.577945:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572538.577967:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572538.578224:0:110198:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc47f05200:110198] 1 exiting: rc = 0 00000100:00100000:0.0:1788572538.578891:0:110197:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc491e9200:110197] 0 exiting: rc = 1 00000100:00100000:1.0:1788572538.582899:0:110199:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc49953400:110199] 2 exiting: rc = 0 00000100:00100000:2.0:1788572538.595114:0:110200:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc47f05600:110200] 0 exiting: rc = 1 00000100:00100000:2.0:1788572538.595208:0:110201:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc47f05800:110201] 1 exiting: rc = 0 00000100:00100000:2.0:1788572538.623543:0:110202:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc47f05400:110202] 2 exiting: rc = 0 00000100:00100000:1.0:1788572538.623860:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623868:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623873:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623878:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623883:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623888:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623893:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623898:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623903:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623907:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623912:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623916:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623921:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623926:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623930:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623935:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623940:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623946:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623962:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623968:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623973:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623978:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623983:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623988:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623994:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.623999:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624004:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624012:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624017:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624022:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624027:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624032:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624037:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624044:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624050:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624055:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624060:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624065:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624069:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624074:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624079:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624084:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624088:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624093:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624097:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624102:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624107:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624112:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624117:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624122:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624127:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624131:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624137:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624143:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624147:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624152:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624157:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624161:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624166:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624171:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624175:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624180:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624184:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624189:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624196:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624201:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624205:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624211:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624226:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624231:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624236:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624241:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624247:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624252:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624256:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624261:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624266:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624270:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624275:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624280:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624284:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624289:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624294:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624299:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624304:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624308:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624313:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624318:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624323:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624328:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624332:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624337:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624342:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624346:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624351:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624355:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624360:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624365:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624369:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624374:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624379:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624383:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624388:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624392:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624397:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624401:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624405:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624410:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624414:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624419:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624423:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624428:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624432:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624437:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624442:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624447:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624451:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624456:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624461:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624465:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624470:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624474:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624478:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624483:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624488:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624492:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624497:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.624501:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572538.760380:0:110708:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:1.0:1788572538.760391:0:110708:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:1.0:1788572538.761101:0:110708:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:1.0:1788572538.761108:0:110708:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572538.761481:0:110708:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572538.761487:0:110708:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572538.762344:0:110708:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572538.762647:0:110708:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572538.763079:0:110708:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572538.763085:0:110708:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:1.0:1788572538.763641:0:110708:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:1.0:1788572538.763647:0:110708:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:0.0:1788572538.764345:0:110708:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:0.0:1788572538.764353:0:110708:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00010000:00010000:0.0:1788572538.766614:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572538.766618:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572538.766620:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572538.767396:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572538.767399:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572538.767400:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572538.768364:0:110708:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:2.0:1788572538.768372:0:110708:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 00010000:00010000:2.0:1788572538.770065:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572538.775850:0:110723:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000100:00100000:0.0:1788572538.776467:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2392:1875449863896704:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788572538.785263:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572538.785283:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:0.0:1788572538.785294:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:0.0:1788572538.785301:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:0.0:1788572538.785307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc67144000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:0.0:1788572538.785310:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788572538.791451:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:0.0:1788572538.791474:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a88e17a8): req@ffff92fc6733b480 x1875449863896832/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.791505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2392:1875449863896704:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788572538.848351:0:110708:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110708:1875449863897344:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572538.848498:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49240 going to sleep for 11 seconds 00000100:00100000:0.0:1788572538.856114:0:110727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110727:1875449863897472:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572538.856273:0:110727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572538.859221:0:110727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110727:1875449863897472:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572538.859269:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.859281:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859283:0:110727:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1788572538.859290:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc5106f000 00000020:00000080:2.0:1788572538.859293:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572538.859306:0:110727:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc5106f000 (ffff92fc509e19c0) 00000020:00000080:2.0:1788572538.859324:0:110727:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:2.0:1788572538.859329:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc5106f000 00000020:00000080:2.0:1788572538.859331:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572538.859584:0:110727:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:2.0:1788572538.859590:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.859595:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859596:0:110727:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1788572538.859599:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.859602:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859603:0:110727:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572538.859606:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc5106f000 00000020:00000080:2.0:1788572538.859608:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572538.859614:0:110727:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc5106f000 (ffff92fc509e3380) 00000020:00000080:2.0:1788572538.859623:0:110727:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:2.0:1788572538.859628:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc5106f000 00000020:00000080:2.0:1788572538.859630:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572538.859852:0:110727:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:2.0:1788572538.859857:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.859861:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859862:0:110727:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572538.859865:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.859868:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859869:0:110727:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1788572538.859873:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1788572538.859874:0:110727:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1788572538.859877:0:110727:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1788572538.859882:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.859884:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859885:0:110727:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1788572538.859887:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.859891:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.859893:0:110727:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1788572538.859896:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572538.859900:0:110727:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572538.859921:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.859924:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572538.859932:0:110727:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc5106f000 (ffff92fc509e4d40) 00000020:00000080:2.0:1788572538.859940:0:110727:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:2.0:1788572538.859945:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.859947:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572538.860084:0:110727:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc6714f800@lustre-MDT0000-mdc-ffff92fc5106f000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572538.866904:0:110727:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc5106f000-1 is initialized 00000040:01000000:2.0:1788572538.868195:0:110727:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc5106f000 ctxt 13 is initialized 00000020:00000080:2.0:1788572538.869868:0:110727:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:2.0:1788572538.870060:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc5106f000 00000020:00000080:2.0:1788572538.870067:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572538.870095:0:110727:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788572538.870132:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.870136:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.870137:0:110727:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788572538.870140:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572538.870143:0:110727:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:2.0:1788572538.870145:0:110727:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572538.870146:0:110727:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572538.870148:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572538.870150:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.870151:0:110727:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572538.870154:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.870158:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.870160:0:110727:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:2.0:1788572538.870162:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572538.870168:0:110727:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572538.870193:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.870196:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572538.870208:0:110727:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc5106f000 (ffff92fc509e0000) 00000020:00000080:2.0:1788572538.870233:0:110727:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:2.0:1788572538.870238:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.870241:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572538.870310:0:110727:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc67148000@lustre-MDT0001-mdc-ffff92fc5106f000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572538.877490:0:110727:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc5106f000-2 is initialized 00000040:01000000:2.0:1788572538.877748:0:110727:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc5106f000 ctxt 13 is initialized 00000020:00000080:2.0:1788572538.880463:0:110727:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:2.0:1788572538.880480:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc5106f000 00000020:00000080:2.0:1788572538.880484:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572538.880511:0:110727:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:2.0:1788572538.880544:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.880548:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.880550:0:110727:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:2.0:1788572538.880553:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572538.880555:0:110727:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:2.0:1788572538.880557:0:110727:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572538.880559:0:110727:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572538.880560:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572538.880563:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.880564:0:110727:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572538.880567:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572538.880572:0:110727:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572538.880575:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.880577:0:110727:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788572538.880579:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572538.880585:0:110727:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572538.880612:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.880615:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572538.880626:0:110727:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc5106f000 (ffff92fc48ccb380) 00000020:00000080:2.0:1788572538.880640:0:110727:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788572538.880646:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.880648:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572538.883748:0:110727:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc6714a800@lustre-OST0000-osc-ffff92fc5106f000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572538.884163:0:110727:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc5106f000-3 is initialized 00000020:00000080:2.0:1788572538.884407:0:110727:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:2.0:1788572538.884422:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc5106f000 00000020:00000080:2.0:1788572538.884427:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572538.884448:0:110727:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788572538.884456:0:110727:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc5106f000: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788572538.884461:0:110727:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788572538.884473:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.884477:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.884479:0:110727:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788572538.884481:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572538.884484:0:110727:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:2.0:1788572538.884486:0:110727:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572538.884487:0:110727:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572538.884489:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572538.884491:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.884493:0:110727:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572538.884495:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572538.884500:0:110727:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:2.0:1788572538.884503:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.884504:0:110727:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788572538.884507:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572538.884512:0:110727:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572538.884532:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.884535:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572538.884543:0:110727:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc5106f000 (ffff92fc48cccd40) 00000020:00000080:2.0:1788572538.884556:0:110727:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:2.0:1788572538.884562:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc5106f000 00000020:00000080:2.0:1788572538.884564:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572538.884624:0:110727:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc51d2b800@lustre-OST0001-osc-ffff92fc5106f000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572538.885272:0:110727:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc5106f000-4 is initialized 00000020:00000080:2.0:1788572538.885465:0:110727:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:2.0:1788572538.885474:0:110727:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc5106f000 00000020:00000080:2.0:1788572538.885477:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572538.885488:0:110727:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1788572538.885494:0:110727:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc5106f000: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1788572538.885497:0:110727:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1788572538.885504:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.885507:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.885509:0:110727:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1788572538.885512:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.885515:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.885517:0:110727:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:2.0:1788572538.885520:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1788572538.885521:0:110727:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1788572538.885524:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572538.885526:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.885527:0:110727:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:2.0:1788572538.885569:0:110727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110727:1875449863897600:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572538.885675:0:110727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572538.898567:0:110727:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110727:1875449863897600:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572538.898612:0:110727:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572538.898621:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572538.898624:0:110727:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572538.898628:0:110727:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572538.898667:0:110727:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110727:1875449863897728:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572538.898778:0:110727:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187240 going to sleep for 11 seconds 00000020:01000000:0.0:1788572538.901471:0:110708:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:0.0:1788572538.901490:0:110708:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572538.901507:0:110708:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572538.901533:0:110708:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572538.901541:0:110708:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc5106f000 from 1 10000000:01000000:0.0:1788572538.901543:0:110708:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572538.901581:0:110708:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572538.901601:0:110708:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572538.901610:0:110708:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572538.901634:0:110708:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110708:1875449863897856:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572538.901701:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572538.973226:0:110733:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:ca18c2b9-77d1-4079-968a-d172b2f6d47e:110733:1875449863898624:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572538.973278:0:110733:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572538.973287:0:110733:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572538.973289:0:110733:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:1.0:1788572538.973293:0:110733:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:0.0:1788572538.973479:0:110708:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:0.0:1788572538.973498:0:110708:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572538.973522:0:110708:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572538.973534:0:110708:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:0.0:1788572538.973553:0:110708:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572538.973579:0:110708:0:(genops.c:1385:class_connect()) connect: client 1190a212-6055-4f84-a4ca-9a6da51fdc58, cookie 0x58a604dcef352861 00800000:01000000:0.0:1788572538.973600:0:110708:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 1190a212-6055-4f84-a4ca-9a6da51fdc58 to lustre-clilmv-ffff92fc5106f000 00800000:01000000:0.0:1788572538.973608:0:110708:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc5106f000(1190a212-6055-4f84-a4ca-9a6da51fdc58) - lustre-MDT0000_UUID, 1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:0.0:1788572538.973619:0:110708:0:(genops.c:1385:class_connect()) connect: client 1190a212-6055-4f84-a4ca-9a6da51fdc58, cookie 0x58a604dcef352868 00000100:00080000:0.0:1788572538.973628:0:110708:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc5106f000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572538.973635:0:110708:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc5106f000: import ffff92fc6714f800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572538.973649:0:110708:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572538.973674:0:110708:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449863898752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572538.973702:0:110708:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 1190a212-6055-4f84-a4ca-9a6da51fdc58->lustre-MDT0000_UUID 00800000:01000000:0.0:1788572538.973917:0:110708:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc5106f000(1190a212-6055-4f84-a4ca-9a6da51fdc58) successfully (5) 00800000:00000004:0.0:1788572538.973921:0:110708:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc5106f000: age 3294, max_age 0 00800000:01000000:0.0:1788572538.975853:0:110708:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc5106f000(1190a212-6055-4f84-a4ca-9a6da51fdc58) - lustre-MDT0001_UUID, 1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:0.0:1788572538.975879:0:110708:0:(genops.c:1385:class_connect()) connect: client 1190a212-6055-4f84-a4ca-9a6da51fdc58, cookie 0x58a604dcef35286f 00000100:00080000:0.0:1788572538.975893:0:110708:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc5106f000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572538.975903:0:110708:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc5106f000: import ffff92fc67148000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572538.975917:0:110708:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572538.975946:0:110708:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449863899008/t0(0) o38->lustre-MDT0001-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572538.975985:0:110708:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 1190a212-6055-4f84-a4ca-9a6da51fdc58->lustre-MDT0001_UUID 00800000:01000000:0.0:1788572538.976233:0:110708:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc5106f000(1190a212-6055-4f84-a4ca-9a6da51fdc58) successfully (5) 00800000:00000004:0.0:1788572538.976238:0:110708:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc5106f000: age 3294, max_age 0 00000080:00000004:0.0:1788572538.976272:0:110708:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc5106f000: age 3294, max_age 4293 00800000:00000004:0.0:1788572538.976278:0:110708:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc5106f000: age 3294, max_age 4293 00000100:00080000:0.0:1788572538.976299:0:110708:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704b100 x1875449863899264/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.976309:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322900 going to sleep for 0 seconds 00000100:00080000:0.0:1788572538.976353:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704b800 x1875449863898880/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572538.976366:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:0.0:1788572538.976370:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704b480 x1875449863899136/t0(0) o41->lustre-MDT0001-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572538.992498:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc6704b480 x1875449863899136/t0(0) o41->lustre-MDT0001-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572538.992518:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863899008:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572538.992539:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5106f000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572538.992545:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc5106f000: connect to target with instance 1 00000100:00080000:1.0:1788572538.992550:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc5106f000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:1.0:1788572538.992556:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:1.0:1788572538.992558:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc6714f800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788572538.992841:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000075a61624): req@ffff92fc6704b100 x1875449863899264/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572538.992858:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc6704b800 x1875449863898880/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572538.992866:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863898752:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572538.992978:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572539.000286:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:1190a212-6055-4f84-a4ca-9a6da51fdc58:0:1875449863899136:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572539.000306:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:1190a212-6055-4f84-a4ca-9a6da51fdc58:0:1875449863898880:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572539.003961:0:110708:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899264:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:2.0:1788572539.003997:0:110708:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc5106f000: update ffff92fc509e5198 cache blocks 317013/353529 objects 1023703/1024000 00000080:00000004:2.0:1788572539.004007:0:110708:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc5106f000: update ffff92fc509e37d8 cache blocks 317013/353529 objects 1023703/1024000 00000100:00100000:2.0:1788572539.004057:0:110708:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899392:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:2.0:1788572539.004139:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572539.015051:0:110708:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899392:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:2.0:1788572539.015087:0:110708:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:2.0:1788572539.015104:0:110708:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:2.0:1788572539.015108:0:110708:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:2.0:1788572539.015126:0:110708:0:(genops.c:1385:class_connect()) connect: client 1190a212-6055-4f84-a4ca-9a6da51fdc58, cookie 0x58a604dcef352876 00000020:00000080:2.0:1788572539.015152:0:110708:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef35287d 00000100:00080000:2.0:1788572539.015159:0:110708:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc5106f000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572539.015170:0:110708:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc5106f000: import ffff92fc6714a800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572539.015182:0:110708:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572539.015208:0:110708:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449863899520/t0(0) o8->lustre-OST0000-osc-ffff92fc5106f000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572539.015254:0:110708:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 1190a212-6055-4f84-a4ca-9a6da51fdc58->lustre-OST0000_UUID 00020000:01000000:2.0:1788572539.015274:0:110708:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc5106f000) active 00000020:00000080:2.0:1788572539.015301:0:110708:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352884 00000100:00080000:2.0:1788572539.015306:0:110708:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc5106f000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572539.015312:0:110708:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc5106f000: import ffff92fc51d2b800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572539.015318:0:110708:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:2.0:1788572539.015330:0:110708:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863899648/t0(0) o8->lustre-OST0001-osc-ffff92fc5106f000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572539.015344:0:110708:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 1190a212-6055-4f84-a4ca-9a6da51fdc58->lustre-OST0001_UUID 00020000:01000000:2.0:1788572539.015349:0:110708:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc5106f000) active 00000100:00100000:2.0:1788572539.015389:0:110708:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899776:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788572539.015465:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187180 going to sleep for 11 seconds 00000100:00100000:2.0:1788572539.023014:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc5106f000: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572539.023033:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc5106f000: connect to target with instance 1 00000100:00080000:2.0:1788572539.023044:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc5106f000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572539.023057:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572539.023061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc6714a800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572539.023068:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1788572539.023073:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572539.023078:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572539.023095:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863899520:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572539.023113:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc5106f000: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572539.023117:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc5106f000: connect to target with instance 1 00000100:00080000:2.0:1788572539.023123:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc5106f000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572539.023127:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788572539.023130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2b800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572539.023132:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:2.0:1788572539.023134:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572539.023136:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572539.023140:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863899648:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572539.023308:0:110708:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899776:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572539.023365:0:110708:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899904:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572539.023474:0:110708:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322900 going to sleep for 11 seconds 00000100:00100000:0.0:1788572539.025632:0:110708:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:1190a212-6055-4f84-a4ca-9a6da51fdc58:110708:1875449863899904:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788572539.025854:0:110708:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665edb88) 00000080:00200000:0.0:1788572539.025863:0:110708:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:0.0:1788572539.025933:0:110708:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:0.0:1788572539.025937:0:110708:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00010000:00010000:1.0:1788572539.338145:0:110764:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:1.0:1788572539.338194:0:110764:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc5106f000 00010000:00010000:1.0:1788572539.338239:0:110764:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc5106f000 00010000:00010000:1.0:1788572539.338520:0:110764:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc5106f000 00010000:00010000:1.0:1788572539.338556:0:110764:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc5106f000 00010000:00010000:0.0:1788572539.781239:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572540.804905:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572541.831792:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788572542.670554:0:110913:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-sec test 59c: MDT migrate of encrypted files without key ========================================================== 21:42:19 (1788572539) 00010000:00010000:0.0:1788572542.854022:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572543.662627:0:110948:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665edb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572543.662897:0:110948:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665edb88),name=/ 00000002:00010000:1.0:1788572543.662923:0:110948:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572543.662986:0:110948:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110948 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.662997:0:110948:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110948 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.663007:0:110948:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110948 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572543.663034:0:110948:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job lsof:1190a212-6055-4f84-a4ca-9a6da51fdc58:110948:1875449863900032:192.168.202.132@tcp:101:lsof.0 00000100:00100000:1.0:1788572543.663165:0:110948:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48e0e000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572543.670242:0:110948:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job lsof:1190a212-6055-4f84-a4ca-9a6da51fdc58:110948:1875449863900032:192.168.202.132@tcp:101:lsof.0 00010000:00010000:1.0:1788572543.670268:0:110948:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670281:0:110948:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670294:0:110948:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670309:0:110948:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670322:0:110948:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670330:0:110948:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670337:0:110948:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572543.670350:0:110948:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b800 x1875449863900032/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572559 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572543.670364:0:110948:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670376:0:110948:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc70727c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572543.670383:0:110948:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572543.670410:0:110948:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572543.670415:0:110948:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665edb88) for lock 0x58a604dcef35288b 00000080:00010000:1.0:1788572543.670420:0:110948:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665edb88) 00000080:00010000:1.0:1788572543.670423:0:110948:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35288b from it ffff9e03821bbcb8 00010000:00010000:1.0:1788572543.670427:0:110948:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572543.670435:0:110948:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572543.883329:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572544.517135:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2395:1875449863900160:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.517295:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900416:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.517332:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900672:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.517360:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572544.517367:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900288:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.517389:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900544:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.531437:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900288:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.531454:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900544:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.531464:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900416:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572544.531475:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:1190a212-6055-4f84-a4ca-9a6da51fdc58:2395:1875449863900672:192.168.202.132@tcp:400:kworker.0 00010000:00010000:0.0:1788572544.901232:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572545.924888:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572546.058587:0:110956:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665edb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572546.058599:0:110956:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665edb88),name=/ 00000002:00010000:2.0:1788572546.058627:0:110956:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572546.058648:0:110956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572546.058665:0:110956:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572546.058673:0:110956:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572546.058995:0:110956:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665edb88) 00000080:00010000:2.0:1788572546.059000:0:110956:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665edb88) for lock 0x58a604dcef35288b 00000080:00010000:2.0:1788572546.059010:0:110956:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35288b from it ffff9e03815afcb8 00010000:00010000:2.0:1788572546.059015:0:110956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572546.059026:0:110956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200008:2.0:1788572546.059253:0:110956:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665edb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572546.059260:0:110956:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665edb88),name=/ 00000002:00010000:2.0:1788572546.059270:0:110956:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572546.059279:0:110956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572546.059288:0:110956:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572546.059296:0:110956:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572546.059300:0:110956:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665edb88) 00000080:00010000:2.0:1788572546.059303:0:110956:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665edb88) for lock 0x58a604dcef35288b 00000080:00010000:2.0:1788572546.059307:0:110956:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35288b from it ffff9e03815afcb8 00010000:00010000:2.0:1788572546.059311:0:110956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572546.059319:0:110956:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106f000 lock: ffff92fc702f4600/0x58a604dcef35288b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792afe expref: -99 pid: 110948 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572546.076914:0:110956:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665edb88) 00000080:00200000:2.0:1788572546.076951:0:110956:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc5106f000 (sb ffff92fc5106f000) 00010000:00010000:2.0:1788572546.076994:0:110956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077007:0:110956:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077014:0:110956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077021:0:110956:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077049:0:110956:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572546.077056:0:110956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077063:0:110956:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077069:0:110956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077074:0:110956:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077098:0:110956:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1788572546.077106:0:110956:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077112:0:110956:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077123:0:110956:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077129:0:110956:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.077136:0:110956:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:2.0:1788572546.077139:0:110956:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:2.0:1788572546.077601:0:110956:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc69d44000 active: 1 00020000:01000000:2.0:1788572546.077605:0:110956:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc69d46000 active: 1 00020000:01000000:2.0:1788572546.077607:0:110956:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788572546.077611:0:110956:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc5106f000 00020000:01000000:2.0:1788572546.077612:0:110956:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc5106f000: disconnecting target lustre-OST0001-osc-ffff92fc5106f000 00000100:00080000:2.0:1788572546.077622:0:110956:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 1190a212-6055-4f84-a4ca-9a6da51fdc58->lustre-OST0001_UUID 00000100:00080000:2.0:1788572546.077713:0:110956:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2b800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:1.0:1788572546.077897:0:110720:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788572546.077905:0:110720:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.077913:0:110720:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204800 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572546.077917:0:110720:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.077925:0:110720:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572546.077931:0:110720:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572546.077937:0:110720:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572546.077940:0:110720:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:1.0:1788572546.077942:0:110720:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:1.0:1788572546.077969:0:110720:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.077976:0:110720:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572546.077996:0:110720:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078003:0:110720:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352822 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792abf expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078018:0:110720:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788572546.078022:0:110720:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078028:0:110720:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2200 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572546.078031:0:110720:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078038:0:110720:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572546.078044:0:110720:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572546.078049:0:110720:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572546.078051:0:110720:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:1.0:1788572546.078053:0:110720:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:1.0:1788572546.078065:0:110720:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078070:0:110720:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572546.078076:0:110720:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078082:0:110720:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352853 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792ac6 expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078093:0:110720:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572546.078096:0:110720:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078102:0:110720:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204200 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:1.0:1788572546.078105:0:110720:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078112:0:110720:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572546.078118:0:110720:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572546.078122:0:110720:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788572546.078124:0:110720:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:1.0:1788572546.078126:0:110720:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:1.0:1788572546.078137:0:110720:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078143:0:110720:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572546.078147:0:110720:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078153:0:110720:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70204200/0x58a604dcef35285a lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792acd expref: -99 pid: 110708 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572546.078162:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572546.078164:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572546.078166:0:110720:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572546.078197:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2394:1875449863900928:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572546.078291:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2394:1875449863901184:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572546.078321:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572546.078327:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2394:1875449863901056:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572546.079209:0:110719:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572546.088404:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2394:1875449863901056:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572546.088426:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2394:1875449863900928:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572546.088437:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2394:1875449863901184:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:2.0:1788572546.091783:0:110956:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572546.091788:0:110956:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc5106f000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572546.091793:0:110956:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572546.093334:0:110956:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352884 00000020:00000080:2.0:1788572546.093361:0:110956:0:(genops.c:885:class_export_put()) final put ffff92fc69d46000/LOV_OSC_UUID 00000020:00000080:1.0:1788572546.094181:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51d2b800 for lustre-OST0001-osc-ffff92fc5106f000 00000020:00000080:1.0:1788572546.094199:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc6714f000/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:1.0:1788572546.094201:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6714f000/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-OST0001-osc-ffff92fc5106f000 00000020:01000000:1.0:1788572546.094209:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc5106f000 (1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000100:00100000:2.0:1788572546.094435:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863901312:192.168.202.132@tcp:9:umount.0 00000100:00080000:2.0:1788572546.097271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc6714a800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572546.097300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863901312:192.168.202.132@tcp:9:umount.0 00000020:00000080:1.0:1788572546.097960:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d44000/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc5106f000 00000020:00000080:1.0:1788572546.101104:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc6714a800 for lustre-OST0000-osc-ffff92fc5106f000 00000020:00000080:1.0:1788572546.101119:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc6714d000/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:1.0:1788572546.101122:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6714d000/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-OST0000-osc-ffff92fc5106f000 00000020:01000000:1.0:1788572546.101128:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc5106f000 (1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:00000080:2.0:1788572546.101211:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc69d42000/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-clilov-ffff92fc5106f000 00000100:00100000:2.0:1788572546.102688:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1190a212-6055-4f84-a4ca-9a6da51fdc58:2392:1875449863901568:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572546.122136:0:110956:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572546.122141:0:110956:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5106f000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572546.122155:0:110956:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67339c00 x1875449863901568/t4294983749(4294983749) o39->lustre-MDT0000-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572556 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572546.122180:0:110956:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572546.129704:0:110956:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352868 00000020:00000080:0.0:1788572546.129932:0:110956:0:(genops.c:885:class_export_put()) final put ffff92fc492e2000/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:1.0:1788572546.129995:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492e2000/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-MDT0000-mdc-ffff92fc5106f000 00000100:00080000:0.0:1788572546.130020:0:110956:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 1190a212-6055-4f84-a4ca-9a6da51fdc58->lustre-MDT0001_UUID 00000100:00080000:0.0:1788572546.138120:0:110956:0:(import.c:65:import_set_state_nolock()) ffff92fc67148000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:0.0:1788572546.141617:0:110956:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572546.141622:0:110956:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5106f000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572546.141633:0:110956:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6704ad80 x1875449863901696/t4294976751(4294976751) o39->lustre-MDT0001-mdc-ffff92fc5106f000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572556 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572546.141657:0:110956:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572546.145135:0:110956:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef35286f 00000020:00000080:0.0:1788572546.145152:0:110956:0:(genops.c:885:class_export_put()) final put ffff92fc50c38000/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:1.0:1788572546.145288:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc50c38000/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-MDT0001-mdc-ffff92fc5106f000 00000020:00000080:1.0:1788572546.145325:0:110956:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352861 00000020:00000080:1.0:1788572546.145329:0:110956:0:(genops.c:885:class_export_put()) final put ffff92fc492e1800/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:01000000:1.0:1788572546.145399:0:110956:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc5106f000 (flags='') 00000020:00000080:1.0:1788572546.145410:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572546.145417:0:110956:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc5106f000: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572546.145422:0:110956:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc509e19c0) has no exports 00000020:00000080:1.0:1788572546.145480:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572546.145486:0:110956:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:00000080:1.0:1788572546.145489:0:110956:0:(genops.c:885:class_export_put()) final put ffff92fc6714b800/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:1.0:1788572546.145492:0:110956:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6714b800/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-clilov-ffff92fc5106f000 00000020:01000000:1.0:1788572546.145497:0:110956:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc5106f000 (1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:1.0:1788572546.145648:0:110956:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc5106f000 (flags='') 00000020:00000080:1.0:1788572546.145652:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572546.145655:0:110956:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc5106f000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572546.145658:0:110956:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc509e3380) has no exports 00000020:00000080:1.0:1788572546.145698:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572546.145701:0:110956:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:1.0:1788572546.145704:0:110956:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc5106f000 (flags='') 00000020:00000080:1.0:1788572546.145707:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572546.145709:0:110956:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc5106f000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572546.145711:0:110956:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc509e4d40) has no exports 00000008:01000000:1.0:1788572546.150526:0:110956:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc5106f000: client import never connected 00000100:00080000:1.0:1788572546.150531:0:110956:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572546.150535:0:110956:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5106f000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572546.150543:0:110956:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572546.154352:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572546.154364:0:110956:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc5106f000 (uuid 1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000020:01000000:1.0:1788572546.154369:0:110956:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc5106f000 (flags='') 00000020:00000080:1.0:1788572546.154373:0:110956:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572546.154378:0:110956:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc5106f000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572546.154382:0:110956:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc509e0000) has no exports 00000020:00000080:2.0:1788572546.163986:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc67148000 for lustre-MDT0001-mdc-ffff92fc5106f000 00000020:00000080:2.0:1788572546.163996:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc6714c000/1190a212-6055-4f84-a4ca-9a6da51fdc58 00000020:00000080:2.0:1788572546.163999:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6714c000/1190a212-6055-4f84-a4ca-9a6da51fdc58 for lustre-MDT0001-mdc-ffff92fc5106f000 00000020:01000000:2.0:1788572546.164009:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc5106f000 (1190a212-6055-4f84-a4ca-9a6da51fdc58) 00000100:00080000:0.0:1788572546.170136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc67144000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:0.0:1788572546.170148:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788572546.170151:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788572546.170152:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:0.0:1788572546.170166:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ca18c2b9-77d1-4079-968a-d172b2f6d47e:2392:1875449863901952:192.168.202.132@tcp:251:umount.0 00000020:00000080:0.0:1788572546.182341:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67144800/ca18c2b9-77d1-4079-968a-d172b2f6d47e for MGC192.168.202.132@tcp 00000020:00000080:0.0:1788572546.182361:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc67144000 for MGC192.168.202.132@tcp 00000020:00000080:0.0:1788572546.182371:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc5106b000/ca18c2b9-77d1-4079-968a-d172b2f6d47e 00000020:00000080:0.0:1788572546.182374:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5106b000/ca18c2b9-77d1-4079-968a-d172b2f6d47e for MGC192.168.202.132@tcp 00000020:01000000:0.0:1788572546.182377:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (ca18c2b9-77d1-4079-968a-d172b2f6d47e) 00010000:00010000:1.0:1788572546.201141:0:110720:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00000100:00100000:0.0:1788572546.210636:0:110709:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc47f05800:110709] 0 exiting: rc = 1 00000100:00100000:0.0:1788572546.210767:0:110710:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc4929c400:110710] 1 exiting: rc = 0 00000100:00100000:1.0:1788572546.215482:0:110714:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc4929c200:110714] 1 exiting: rc = 0 00000100:00100000:0.0:1788572546.218215:0:110713:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc4929ce00:110713] 0 exiting: rc = 1 00000100:00100000:0.0:1788572546.218305:0:110715:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc4a1bd800:110715] 2 exiting: rc = 0 00000100:00100000:2.0:1788572546.229023:0:110718:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc47f13a00:110718] 2 exiting: rc = 0 00000100:00100000:2.0:1788572546.229445:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229454:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229459:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229465:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229471:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229476:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229481:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229487:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229491:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229496:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229501:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229507:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229512:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229518:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229523:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229528:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229534:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229539:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229544:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229549:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229554:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229560:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229565:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229571:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229576:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229581:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229585:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229591:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229596:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229602:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229607:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229611:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229616:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229622:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229628:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229634:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229640:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229646:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229652:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229657:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229662:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229668:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229674:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229696:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229702:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229707:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229712:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229718:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229748:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229754:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229760:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229765:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229771:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229777:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229782:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229787:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229793:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229798:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229802:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229807:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229812:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229817:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229823:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229827:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229834:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229839:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229844:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229849:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229855:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229861:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229867:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229873:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229879:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229884:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229890:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229895:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229901:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229906:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229911:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229917:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229922:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229928:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229933:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229939:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229944:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229950:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229955:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229960:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229966:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229972:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229977:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229981:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229986:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229991:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.229996:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230001:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230006:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230012:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230017:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230024:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230029:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230034:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230039:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230043:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230049:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230053:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230058:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230064:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230068:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230073:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230077:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230082:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230087:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230092:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230097:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230103:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230108:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230112:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230117:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230121:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230126:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230131:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230136:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230141:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230146:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230151:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230156:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572546.230162:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:0.0:1788572546.422799:0:110969:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:0.0:1788572546.422805:0:110969:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:0.0:1788572546.422806:0:110969:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:0.0:1788572546.422808:0:110969:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock,test_dummy_encryption 00000080:01200004:0.0:1788572546.423064:0:110969:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc4536c800 00000080:01000004:0.0:1788572546.423076:0:110969:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:0.0:1788572546.423092:0:110969:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:0.0:1788572546.423122:0:110969:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:0.0:1788572546.423133:0:110969:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc70034d40) 00000020:00000080:0.0:1788572546.423147:0:110969:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:0.0:1788572546.423326:0:110969:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572546.423414:0:110969:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572546.423626:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:0.0:1788572546.423633:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:1.0:1788572546.424468:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:1.0:1788572546.424476:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:0.0:1788572546.425108:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:0.0:1788572546.425116:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572546.425844:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572546.426091:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:0.0:1788572546.427211:0:110969:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572546.427650:0:110969:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572546.428269:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:0.0:1788572546.428276:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788572546.431307:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788572546.431322:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:0.0:1788572546.432362:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:0.0:1788572546.432371:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788572546.433252:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788572546.433261:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:0.0:1788572546.434143:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:0.0:1788572546.434152:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:0.0:1788572546.434644:0:110969:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:0.0:1788572546.434650:0:110969:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:2.0:1788572546.436116:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572546.436122:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572546.436124:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572546.436463:0:110981:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572546.436466:0:110981:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572546.436475:0:110981:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:0.0:1788572546.437405:0:110969:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc42c0f800@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572546.437644:0:110969:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:0.0:1788572546.437653:0:110969:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 00010000:00010000:2.0:1788572546.442479:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:00000080:0.0:1788572546.446105:0:110969:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid 4761e684-853a-44f0-be14-fa356e387a10) 00000020:00000080:0.0:1788572546.446135:0:110969:0:(genops.c:1385:class_connect()) connect: client 4761e684-853a-44f0-be14-fa356e387a10, cookie 0x58a604dcef352899 00000100:00080000:0.0:1788572546.446151:0:110969:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572546.446163:0:110969:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc42c0f800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572546.446210:0:110969:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863902080/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572546.447259:0:110969:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 4761e684-853a-44f0-be14-fa356e387a10->MGS 00000080:00200000:0.0:1788572546.447290:0:110969:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc4536c800 (sb ffff92fc4536c800) 00000080:01000000:0.0:1788572546.447430:0:110969:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock,test_dummy_encryption 00000080:00000004:0.0:1788572546.447433:0:110969:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788572546.447437:0:110969:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:00000004:0.0:1788572546.447439:0:110969:0:(llite_lib.c:1242:ll_options()) next opt=test_dummy_encryption 00000080:01000000:0.0:1788572546.447448:0:110969:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 2a90d0e1-c03b-4477-a84e-0596af2dd806 10000000:01000000:0.0:1788572546.448434:0:110969:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788572546.448438:0:110969:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc4536c800 10000000:01000000:0.0:1788572546.448443:0:110969:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc70034d40 10000000:01000000:0.0:1788572546.448447:0:110969:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572546.448456:0:110969:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc70034d40 from 1 10000000:01000000:0.0:1788572546.448459:0:110969:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572546.448497:0:110969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572546.448509:0:110969:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572546.448520:0:110969:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572546.448544:0:110969:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67049f80 x1875449863902208/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572546.448554:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6038dc00 going to sleep for 0 seconds 00000100:00100000:3.0:1788572546.492321:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863902592:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572546.492387:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863902720:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572546.492490:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6038dc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.496358:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863902720:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572546.497149:0:110988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110988:1875449863902848:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572546.497314:0:110988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572546.501438:0:110988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110988:1875449863902848:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572546.501489:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572546.501501:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572546.501503:0:110988:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:0.0:1788572546.501511:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc4536c800 00000020:00000080:0.0:1788572546.501514:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572546.501528:0:110988:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc4536c800 (ffff92fc70033380) 00000020:00000080:0.0:1788572546.501548:0:110988:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:0.0:1788572546.501555:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc4536c800 00000020:00000080:0.0:1788572546.501558:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572546.502136:0:110988:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:0.0:1788572546.502142:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572546.502148:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572546.502150:0:110988:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:0.0:1788572546.502156:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572546.502159:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572546.502161:0:110988:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572546.502164:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc4536c800 00000020:00000080:0.0:1788572546.502167:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572546.502173:0:110988:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc4536c800 (ffff92fc700319c0) 00000020:00000080:0.0:1788572546.502193:0:110988:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:0.0:1788572546.502199:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc4536c800 00000020:00000080:0.0:1788572546.502201:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:1.0:1788572546.510659:0:110988:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:1.0:1788572546.510689:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.510697:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.510699:0:110988:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:1.0:1788572546.510703:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572546.510708:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.510709:0:110988:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:1.0:1788572546.510713:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:1.0:1788572546.510715:0:110988:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:1.0:1788572546.510744:0:110988:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:1.0:1788572546.510752:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.510755:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.510757:0:110988:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:1.0:1788572546.510759:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572546.510762:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.510764:0:110988:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:1.0:1788572546.510766:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572546.510772:0:110988:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572546.510801:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.510804:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572546.510818:0:110988:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc4536c800 (ffff92fc49933380) 00000020:00000080:1.0:1788572546.510839:0:110988:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:1.0:1788572546.510844:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.510846:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572546.511132:0:110988:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5838d800@lustre-MDT0000-mdc-ffff92fc4536c800: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572546.513351:0:110988:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc4536c800-1 is initialized 00000040:01000000:1.0:1788572546.513604:0:110988:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc4536c800 ctxt 13 is initialized 00000020:00000080:1.0:1788572546.514163:0:110988:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:1.0:1788572546.514177:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc4536c800 00000020:00000080:1.0:1788572546.514193:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:1.0:1788572546.514222:0:110988:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:1.0:1788572546.514256:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.514260:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.514262:0:110988:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:1.0:1788572546.514265:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:1.0:1788572546.514268:0:110988:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:1.0:1788572546.514269:0:110988:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572546.514271:0:110988:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572546.514272:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:1.0:1788572546.514275:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.514276:0:110988:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:1.0:1788572546.514278:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572546.514283:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.514285:0:110988:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:1.0:1788572546.514288:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572546.514292:0:110988:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572546.514318:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.514320:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572546.514331:0:110988:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc4536c800 (ffff92fc49934d40) 00000020:00000080:1.0:1788572546.514343:0:110988:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:1.0:1788572546.514349:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.514351:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572546.514430:0:110988:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5106b000@lustre-MDT0001-mdc-ffff92fc4536c800: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572546.521816:0:110988:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc4536c800-2 is initialized 00000040:01000000:1.0:1788572546.523838:0:110988:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc4536c800 ctxt 13 is initialized 00000020:00000080:1.0:1788572546.525433:0:110988:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:1.0:1788572546.525450:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc4536c800 00000020:00000080:1.0:1788572546.525455:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:1.0:1788572546.525481:0:110988:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:1.0:1788572546.525509:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.525513:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.525516:0:110988:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:1.0:1788572546.525519:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:1.0:1788572546.525522:0:110988:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:1.0:1788572546.525523:0:110988:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572546.525525:0:110988:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572546.525527:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:1.0:1788572546.525529:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.525530:0:110988:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:1.0:1788572546.525533:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:1.0:1788572546.525537:0:110988:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:1.0:1788572546.525541:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.525542:0:110988:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:1.0:1788572546.525545:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572546.525551:0:110988:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572546.525579:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.525582:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572546.525592:0:110988:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc4536c800 (ffff92fc499319c0) 00000020:00000080:1.0:1788572546.525608:0:110988:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:1.0:1788572546.525612:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.525615:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572546.528126:0:110988:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc51069800@lustre-OST0000-osc-ffff92fc4536c800: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572546.528382:0:110988:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc4536c800-3 is initialized 00000020:00000080:1.0:1788572546.528627:0:110988:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:1.0:1788572546.528640:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc4536c800 00000020:00000080:1.0:1788572546.528645:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:1.0:1788572546.528667:0:110988:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:1.0:1788572546.528675:0:110988:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc4536c800: Adding target lustre-OST0000_UUID 00020000:01000000:1.0:1788572546.528680:0:110988:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:1.0:1788572546.528693:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.528697:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.528700:0:110988:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:1.0:1788572546.528703:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:1.0:1788572546.528707:0:110988:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:1.0:1788572546.528708:0:110988:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572546.528710:0:110988:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:1.0:1788572546.528711:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:1.0:1788572546.528714:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.528715:0:110988:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:1.0:1788572546.528718:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:1.0:1788572546.528745:0:110988:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:1.0:1788572546.528749:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.528751:0:110988:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:1.0:1788572546.528755:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:1.0:1788572546.528760:0:110988:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:1.0:1788572546.528783:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.528786:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788572546.528796:0:110988:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc4536c800 (ffff92fc49930000) 00000020:00000080:1.0:1788572546.528811:0:110988:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:1.0:1788572546.528817:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc4536c800 00000020:00000080:1.0:1788572546.528819:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:1.0:1788572546.528883:0:110988:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc51d2d000@lustre-OST0001-osc-ffff92fc4536c800: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572546.534298:0:110988:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc4536c800-4 is initialized 00000020:00000080:1.0:1788572546.534531:0:110988:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:1.0:1788572546.534545:0:110988:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc4536c800 00000020:00000080:1.0:1788572546.534549:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:1.0:1788572546.534570:0:110988:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:1.0:1788572546.534579:0:110988:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc4536c800: Adding target lustre-OST0001_UUID 00020000:01000000:1.0:1788572546.534583:0:110988:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:1.0:1788572546.534592:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.534596:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.534598:0:110988:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:1.0:1788572546.534601:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572546.534605:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.534606:0:110988:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:1.0:1788572546.534609:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:1.0:1788572546.534612:0:110988:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:1.0:1788572546.534614:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572546.534616:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572546.534617:0:110988:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:1.0:1788572546.534665:0:110988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110988:1875449863902976:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572546.534790:0:110988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.546391:0:110988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110988:1875449863902976:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572546.546435:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572546.546445:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.546447:0:110988:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:3.0:1788572546.546452:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:3.0:1788572546.546491:0:110988:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110988:1875449863903104:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572546.546570:0:110988:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.554610:0:110988:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110988:1875449863903104:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572546.554653:0:110988:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572546.554661:0:110988:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.554664:0:110988:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:3.0:1788572546.554668:0:110988:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:3.0:1788572546.554807:0:110969:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:3.0:1788572546.554825:0:110969:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572546.554845:0:110969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.554858:0:110969:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572546.554867:0:110969:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc4536c800 from 1 10000000:01000000:3.0:1788572546.554869:0:110969:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788572546.554905:0:110969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.554914:0:110969:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.554923:0:110969:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572546.554945:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903232:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572546.555796:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.559444:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903232:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572546.559475:0:110969:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.559489:0:110969:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572546.560469:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903360:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788572546.560627:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.568162:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903360:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:3.0:1788572546.569207:0:110969:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:3.0:1788572546.569282:0:110969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.569295:0:110969:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572546.569308:0:110969:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc4536c800 from 1 10000000:01000000:3.0:1788572546.569312:0:110969:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788572546.569353:0:110969:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.569362:0:110969:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.569371:0:110969:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572546.569395:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903488:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572546.569476:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.577312:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903488:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572546.577346:0:110969:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572546.577360:0:110969:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572546.577436:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903616:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572546.577507:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.581192:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903616:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572546.581256:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903744:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572546.581339:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.584779:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4761e684-853a-44f0-be14-fa356e387a10:110969:1875449863903744:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572546.585376:0:110993:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110993:1875449863903872:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572546.585524:0:110993:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.588804:0:110993:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110993:1875449863903872:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572546.588855:0:110993:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572546.588870:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.588873:0:110993:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:3.0:1788572546.588880:0:110993:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc4536c800 00000020:00000080:3.0:1788572546.588883:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572546.588892:0:110993:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572546.588894:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.588896:0:110993:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:3.0:1788572546.588899:0:110993:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572546.588901:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.588902:0:110993:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:3.0:1788572546.588905:0:110993:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc4536c800 00000020:00000080:3.0:1788572546.588907:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572546.589142:0:110993:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572546.589147:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.589149:0:110993:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:3.0:1788572546.589152:0:110993:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572546.589155:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.589156:0:110993:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:3.0:1788572546.589160:0:110993:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc4536c800 00000020:00000080:3.0:1788572546.589162:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:3.0:1788572546.589630:0:110993:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 461us 00000100:00100000:3.0:1788572546.589673:0:110993:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110993:1875449863904000:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572546.589812:0:110993:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572546.596894:0:110993:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4761e684-853a-44f0-be14-fa356e387a10:110993:1875449863904000:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572546.596955:0:110993:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572546.596972:0:110993:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572546.596974:0:110993:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:3.0:1788572546.596979:0:110993:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:0.0:1788572546.597273:0:110969:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:0.0:1788572546.597300:0:110969:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572546.597332:0:110969:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572546.597344:0:110969:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:0.0:1788572546.597366:0:110969:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572546.597391:0:110969:0:(genops.c:1385:class_connect()) connect: client 2a90d0e1-c03b-4477-a84e-0596af2dd806, cookie 0x58a604dcef3528e6 00800000:01000000:0.0:1788572546.598110:0:110969:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 2a90d0e1-c03b-4477-a84e-0596af2dd806 to lustre-clilmv-ffff92fc4536c800 00800000:01000000:0.0:1788572546.598119:0:110969:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc4536c800(2a90d0e1-c03b-4477-a84e-0596af2dd806) - lustre-MDT0000_UUID, 2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:0.0:1788572546.598131:0:110969:0:(genops.c:1385:class_connect()) connect: client 2a90d0e1-c03b-4477-a84e-0596af2dd806, cookie 0x58a604dcef3528ed 00000100:00080000:0.0:1788572546.598144:0:110969:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc4536c800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572546.598154:0:110969:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc4536c800: import ffff92fc5838d800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572546.598168:0:110969:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572546.598209:0:110969:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449863904128/t0(0) o38->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572546.598238:0:110969:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-MDT0000_UUID 00800000:01000000:0.0:1788572546.598337:0:110969:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc4536c800(2a90d0e1-c03b-4477-a84e-0596af2dd806) successfully (5) 00800000:00000004:0.0:1788572546.598341:0:110969:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc4536c800: age 3302, max_age 0 00800000:01000000:0.0:1788572546.598376:0:110969:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc4536c800(2a90d0e1-c03b-4477-a84e-0596af2dd806) - lustre-MDT0001_UUID, 2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:0.0:1788572546.598383:0:110969:0:(genops.c:1385:class_connect()) connect: client 2a90d0e1-c03b-4477-a84e-0596af2dd806, cookie 0x58a604dcef3528f4 00000100:00080000:0.0:1788572546.598388:0:110969:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc4536c800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572546.598394:0:110969:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc4536c800: import ffff92fc5106b000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572546.598400:0:110969:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572546.598412:0:110969:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449863904384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572546.598420:0:110969:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-MDT0001_UUID 00800000:01000000:0.0:1788572546.598463:0:110969:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc4536c800(2a90d0e1-c03b-4477-a84e-0596af2dd806) successfully (5) 00800000:00000004:0.0:1788572546.598466:0:110969:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc4536c800: age 3302, max_age 0 00000080:00000004:0.0:1788572546.598483:0:110969:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc4536c800: age 3302, max_age 4301 00800000:00000004:0.0:1788572546.598489:0:110969:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc4536c800: age 3302, max_age 4301 00000100:00080000:0.0:1788572546.598509:0:110969:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67048e00 x1875449863904640/t0(0) o41->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572546.598518:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788572546.598825:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863904128:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572546.598912:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863904384:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572546.606311:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4536c800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572546.606329:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc4536c800: connect to target with instance 1 00000100:00080000:3.0:1788572546.606337:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc4536c800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572546.606350:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:3.0:1788572546.606354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5106b000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:2.0:1788572546.607992:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863904384:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572546.608045:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4536c800: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572546.608053:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc4536c800: connect to target with instance 1 00000100:00080000:2.0:1788572546.608061:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc4536c800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:2.0:1788572546.608071:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:2.0:1788572546.608074:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5838d800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788572546.608490:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f9c19992): req@ffff92fc67048e00 x1875449863904640/t0(0) o41->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572546.608510:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863904128:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788572546.608633:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572546.609234:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2a90d0e1-c03b-4477-a84e-0596af2dd806:2394:1875449863904512:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572546.609278:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788572546.609284:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2a90d0e1-c03b-4477-a84e-0596af2dd806:2394:1875449863904256:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572546.623746:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2a90d0e1-c03b-4477-a84e-0596af2dd806:2394:1875449863904256:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572546.624164:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863904640:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:3.0:1788572546.624194:0:110969:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc4536c800: update ffff92fc499337d8 cache blocks 317013/353529 objects 1023703/1024000 00000080:00000004:3.0:1788572546.624202:0:110969:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc4536c800: update ffff92fc70031e18 cache blocks 317013/353529 objects 1023703/1024000 00000100:00100000:3.0:1788572546.624246:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863904768:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:3.0:1788572546.624322:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572546.630330:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:2a90d0e1-c03b-4477-a84e-0596af2dd806:2394:1875449863904512:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572546.630893:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863904768:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:0.0:1788572546.630913:0:110969:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:0.0:1788572546.630940:0:110969:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:0.0:1788572546.630945:0:110969:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:0.0:1788572546.631663:0:110969:0:(genops.c:1385:class_connect()) connect: client 2a90d0e1-c03b-4477-a84e-0596af2dd806, cookie 0x58a604dcef3528fb 00000020:00000080:0.0:1788572546.631697:0:110969:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352902 00000100:00080000:0.0:1788572546.631706:0:110969:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc4536c800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572546.631716:0:110969:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc4536c800: import ffff92fc51069800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572546.631761:0:110969:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572546.631791:0:110969:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863904896/t0(0) o8->lustre-OST0000-osc-ffff92fc4536c800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572546.631881:0:110969:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-OST0000_UUID 00020000:01000000:0.0:1788572546.631901:0:110969:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc4536c800) active 00000020:00000080:0.0:1788572546.631941:0:110969:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352909 00000100:00080000:0.0:1788572546.631947:0:110969:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc4536c800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572546.631952:0:110969:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc4536c800: import ffff92fc51d2d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572546.631966:0:110969:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572546.631978:0:110969:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863905024/t0(0) o8->lustre-OST0001-osc-ffff92fc4536c800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572546.631992:0:110969:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-OST0001_UUID 00020000:01000000:0.0:1788572546.631997:0:110969:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc4536c800) active 00000080:02000400:0.0:1788572546.632007:0:110969:0:(llite_lib.c:801:client_common_fill_super()) Test dummy encryption mode enabled 00000100:00100000:0.0:1788572546.632040:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863905152:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572546.632247:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572546.634320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863904896:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572546.634423:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863905024:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572546.640443:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4536c800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572546.640458:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc4536c800: connect to target with instance 1 00000100:00080000:0.0:1788572546.640468:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc4536c800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788572546.640479:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1788572546.640483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51069800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:0.0:1788572546.640490:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:0.0:1788572546.640494:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788572546.640498:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:0.0:1788572546.640513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863904896:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572546.640529:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4536c800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572546.640532:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc4536c800: connect to target with instance 1 00000100:00080000:0.0:1788572546.640536:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc4536c800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788572546.640539:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1788572546.640541:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:0.0:1788572546.640545:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:0.0:1788572546.640547:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788572546.640549:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:0.0:1788572546.640553:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863905024:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572546.641153:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863905152:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:2.0:1788572546.641227:0:110969:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863905280:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:2.0:1788572546.641335:0:110969:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572546.643473:0:110969:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:2a90d0e1-c03b-4477-a84e-0596af2dd806:110969:1875449863905280:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:2.0:1788572546.643652:0:110969:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00200000:2.0:1788572546.643662:0:110969:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:2.0:1788572546.643865:0:110969:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:2.0:1788572546.643870:0:110969:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00000080:00200000:0.0:1788572546.798873:0:111011:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:0.0:1788572546.799034:0:111011:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572546.799097:0:111011:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111011 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572546.799109:0:111011:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111011 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572546.799119:0:111011:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111011 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572546.799146:0:111011:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job mkdir:2a90d0e1-c03b-4477-a84e-0596af2dd806:111011:1875449863905408:192.168.202.132@tcp:101:mkdir.0 00000100:00100000:0.0:1788572546.799291:0:111011:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572546.805177:0:111011:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job mkdir:2a90d0e1-c03b-4477-a84e-0596af2dd806:111011:1875449863905408:192.168.202.132@tcp:101:mkdir.0 00010000:00010000:2.0:1788572546.805218:0:111011:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805232:0:111011:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805244:0:111011:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805261:0:111011:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805277:0:111011:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805285:0:111011:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805292:0:111011:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572546.805304:0:111011:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704a680 x1875449863905408/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572562 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572546.805317:0:111011:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805331:0:111011:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572546.805338:0:111011:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572546.805368:0:111011:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572546.805373:0:111011:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352910 00000080:00010000:2.0:1788572546.805377:0:111011:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:2.0:1788572546.805380:0:111011:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352910 from it ffff9e03821abbe0 00010000:00010000:2.0:1788572546.805383:0:111011:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572546.805392:0:111011:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572546.805418:0:111011:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572546.805448:0:111011:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e0908), flags=1536 00000080:00200000:2.0:1788572546.805520:0:111011:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:2.0:1788572546.805538:0:111011:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572546.805548:0:111011:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572546.805559:0:111011:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572546.805565:0:111011:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572546.805570:0:111011:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:2.0:1788572546.805572:0:111011:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352910 00000080:00010000:2.0:1788572546.805576:0:111011:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352910 from it ffff9e03821abce0 00010000:00010000:2.0:1788572546.805579:0:111011:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572546.805586:0:111011:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572546.805595:0:111011:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 3 00000080:00200000:2.0:1788572546.805601:0:111011:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:2.0:1788572546.805608:0:111011:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572546.805614:0:111011:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572546.805622:0:111011:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572546.805628:0:111011:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572546.805631:0:111011:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:2.0:1788572546.805633:0:111011:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352910 00000080:00010000:2.0:1788572546.805637:0:111011:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352910 from it ffff9e03821abdc8 00010000:00010000:2.0:1788572546.805640:0:111011:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572546.805648:0:111011:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572546.805656:0:111011:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 3 00000080:00200000:2.0:1788572546.805662:0:111011:0:(namei.c:2172:do_mkdir()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e0908) 00000100:00100000:2.0:1788572546.839221:0:111011:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mkdir:2a90d0e1-c03b-4477-a84e-0596af2dd806:111011:1875449863905536:192.168.202.132@tcp:700:mkdir.0 00000100:00100000:2.0:1788572546.839346:0:111011:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572546.841687:0:111011:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mkdir:2a90d0e1-c03b-4477-a84e-0596af2dd806:111011:1875449863905536:192.168.202.132@tcp:700:mkdir.0 00010000:00010000:2.0:1788572546.841769:0:111011:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572546.841827:0:111011:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572546.841835:0:111011:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572546.841838:0:111011:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3800/0x58a604dcef352910 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792b4b expref: -99 pid: 111011 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000020:00100000:2.0:1788572546.841864:0:111011:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4536c800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572546.841873:0:111011:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ca80 x1875449863905792/t0(0) o36->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 592/688 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572546.841905:0:111011:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mkdir:2a90d0e1-c03b-4477-a84e-0596af2dd806:111011:1875449863905792:192.168.202.132@tcp:36:mkdir.0 00000100:00100000:2.0:1788572546.841978:0:111011:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572546.846483:0:111011:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mkdir:2a90d0e1-c03b-4477-a84e-0596af2dd806:111011:1875449863905792:192.168.202.132@tcp:36:mkdir.0 00000080:00200000:2.0:1788572546.846540:0:111011:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348) 00000080:00200000:2.0:1788572546.846547:0:111011:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00200000:0.0:1788572547.051991:0:111018:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:0.0:1788572547.052038:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572547.052109:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.052120:0:111018:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.052131:0:111018:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572547.052159:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863905920:192.168.202.132@tcp:101:lfs.0 00000100:00100000:0.0:1788572547.052671:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572547.061533:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863905920:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572547.061563:0:111018:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061575:0:111018:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061588:0:111018:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061601:0:111018:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061616:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061624:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061631:0:111018:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572547.061643:0:111018:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704a300 x1875449863905920/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572547.061658:0:111018:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061673:0:111018:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572547.061680:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572547.061717:0:111018:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572547.061776:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:1.0:1788572547.061784:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:1.0:1788572547.061787:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0381723b58 00010000:00010000:1.0:1788572547.061791:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.061800:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572547.061829:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:1.0:1788572547.061875:0:111018:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e0908), flags=29 00000080:00200000:1.0:1788572547.061881:0:111018:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665e0908), intent=0 00000002:00010000:1.0:1788572547.061906:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59c.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572547.061950:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.062131:0:111018:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.062147:0:111018:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572547.062170:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906048:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572547.062264:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572547.075642:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906048:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572547.075670:0:111018:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075705:0:111018:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075714:0:111018:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075766:0:111018:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075779:0:111018:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075791:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075799:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075811:0:111018:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572547.075824:0:111018:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc67289c00 x1875449863906048/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572547.075835:0:111018:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075846:0:111018:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589935649/1 (0 0) 00000080:00200000:1.0:1788572547.075853:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572547.075887:0:111018:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572547.075891:0:111018:0:(namei.c:147:ll_iget()) got inode: [0x200000421:0x1:0x0](ffff92fc665e5348): rc = 0 00000080:00010000:1.0:1788572547.075903:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5348) for lock 0x58a604dcef35291e 00000080:00010000:1.0:1788572547.075915:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5348) 00000080:00010000:1.0:1788572547.075919:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35291e from it ffff9e0381723a28 00010000:00010000:1.0:1788572547.075923:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.075932:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572547.075976:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348), inode mode 41ed mask 1 00000080:00200000:1.0:1788572547.075994:0:111018:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e5348), flags=13 00000080:00200000:1.0:1788572547.075998:0:111018:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e5348), intent=getattr 00000100:00100000:1.0:1788572547.078554:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906176:192.168.202.132@tcp:900:lfs.0 00000100:00100000:1.0:1788572547.078743:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572547.090467:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906176:192.168.202.132@tcp:900:lfs.0 00000002:00010000:1.0:1788572547.090513:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572547.090571:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.090581:0:111018:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.090592:0:111018:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572547.090614:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906304:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572547.090704:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572547.103756:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906304:192.168.202.132@tcp:101:lfs.0 00010000:00010000:3.0:1788572547.103781:0:111018:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103797:0:111018:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103805:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103814:0:111018:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103823:0:111018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103830:0:111018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f5200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:3.0:1788572547.103835:0:111018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103842:0:111018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103850:0:111018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 5/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103857:0:111018:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572547.103863:0:111018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.103870:0:111018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5200/0x58a604dcef352925 lrc: 0/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572547.103900:0:111018:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff92fc67289c00 x1875449863906304/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572547.103913:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572547.103952:0:111018:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 30 res 8589935649/1 (0 0) 00000080:00200000:3.0:1788572547.104515:0:111018:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:3.0:1788572547.104535:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572547.104546:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572547.104558:0:111018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572547.104566:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572547.104573:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:3.0:1788572547.104577:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:3.0:1788572547.104583:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0381723be8 00010000:00010000:3.0:1788572547.104587:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572547.104596:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572547.104613:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:3.0:1788572547.104622:0:111018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=16408 00000080:00200000:3.0:1788572547.104628:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348), inode mode 41ed mask 1 00000080:00200000:3.0:1788572547.104650:0:111018:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e5348), flags=16392 00000080:00200000:3.0:1788572547.104654:0:111018:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e5348), intent=getattr 00000002:00010000:3.0:1788572547.104702:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572547.104807:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.104815:0:111018:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.104824:0:111018:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572547.104847:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906432:192.168.202.132@tcp:101:lfs.0 00000100:00100000:3.0:1788572547.105213:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572547.115353:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906432:192.168.202.132@tcp:101:lfs.0 00010000:00010000:3.0:1788572547.115381:0:111018:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115396:0:111018:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115403:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115411:0:111018:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115420:0:111018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115426:0:111018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:3.0:1788572547.115431:0:111018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115438:0:111018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115445:0:111018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 5/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115452:0:111018:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572547.115460:0:111018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.115467:0:111018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205400/0x58a604dcef35292c lrc: 0/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572547.115486:0:111018:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff92fc6733bb80 x1875449863906432/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572547.115499:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572547.115528:0:111018:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 30 res 8589935649/1 (0 0) 00000080:00200000:3.0:1788572547.115605:0:111018:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:3.0:1788572547.115621:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572547.115630:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572547.115642:0:111018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572547.115649:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572547.115654:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:3.0:1788572547.115658:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:3.0:1788572547.115664:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0381723b58 00010000:00010000:3.0:1788572547.115668:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572547.118821:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572547.118848:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:3.0:1788572547.118864:0:111018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=13 00000080:00200008:3.0:1788572547.120365:0:111018:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5348), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572547.120381:0:111018:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348),name=d59c.sanity-sec 00000002:00010000:3.0:1788572547.120416:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572547.120429:0:111018:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 30 res 8589935649/1 (0 0) 00010000:00010000:3.0:1788572547.120484:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.120494:0:111018:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.120505:0:111018:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572547.120530:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906560:192.168.202.132@tcp:101:lfs.0 00000100:00100000:3.0:1788572547.120619:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572547.132431:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906560:192.168.202.132@tcp:101:lfs.0 00010000:00010000:3.0:1788572547.132458:0:111018:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132475:0:111018:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132484:0:111018:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132495:0:111018:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132511:0:111018:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132524:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132532:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132539:0:111018:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572547.132551:0:111018:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733bb80 x1875449863906560/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572547.132564:0:111018:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132574:0:111018:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589935649/1 (0 0) 00000080:00200000:3.0:1788572547.132581:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572547.132602:0:111018:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572547.132608:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5348) for lock 0x58a604dcef352933 00000080:00010000:3.0:1788572547.132615:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5348) 00000080:00010000:3.0:1788572547.132618:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352933 from it ffff9e0381723cb8 00010000:00010000:3.0:1788572547.132621:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572547.132633:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572547.133600:0:111018:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e0908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572547.133606:0:111018:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:3.0:1788572547.133625:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572547.133637:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572547.133650:0:111018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572547.133656:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572547.133661:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:3.0:1788572547.133664:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:3.0:1788572547.133669:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0381723cb8 00010000:00010000:3.0:1788572547.136511:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572547.136530:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572547.138200:0:111018:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:2.0:1788572547.138231:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572547.138252:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572547.138268:0:111018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572547.138278:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572547.138286:0:111018:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:2.0:1788572547.138289:0:111018:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:2.0:1788572547.138298:0:111018:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0381723bf8 00010000:00010000:2.0:1788572547.138302:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572547.138310:0:111018:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572547.138326:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572547.138334:0:111018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=267 00000080:00200000:2.0:1788572547.138344:0:111018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348), inode mode 41ed mask 44 00000080:00200008:2.0:1788572547.138519:0:111018:0:(file.c:1090:ll_file_open()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5348), flags 300000 00000002:00010000:2.0:1788572547.138540:0:111018:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:2.0:1788572547.138596:0:111018:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.138604:0:111018:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572547.138617:0:111018:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4536c800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572547.138630:0:111018:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d880 x1875449863906816/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572547.138639:0:111018:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572547.138672:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906816:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572547.139568:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.159548:0:111018:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4536c800: committing for last_committed 4294983751 gen 1 00000100:00100000:2.0:1788572547.159579:0:111018:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617d880 x1875449863906816/t4294983751(4294983751) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572563 ref 3 fl Rpc:RPQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572547.159593:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863906816:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572547.159611:0:111018:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159624:0:111018:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159630:0:111018:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159638:0:111018:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159646:0:111018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159652:0:111018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367800 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572547.159656:0:111018:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159664:0:111018:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159695:0:111018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159703:0:111018:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572547.159712:0:111018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.159718:0:111018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35293a lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111018 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572547.159779:0:111018:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617d880 x1875449863906816/t4294983751(4294983751) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572563 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572547.159793:0:111018:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d880 x1875449863906816/t4294983751(4294983751) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572563 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572547.159807:0:111018:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572547.159831:0:111018:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572547.159852:0:111018:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d880 x1875449863906816/t4294983751(4294983751) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572563 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572547.160407:0:111018:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348) cmd=c03066f0 arg=7ffdee9c8de0 00000080:00200000:2.0:1788572547.160426:0:111018:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348) name=subdir stripe_offset=0 stripe_count=0, hash_type=0 00010000:00010000:2.0:1788572547.160483:0:111018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572547.160503:0:111018:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572547.160549:0:111018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572547.160557:0:111018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572547.160560:0:111018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70205000/0x58a604dcef352933 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792bc2 expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572547.160581:0:111018:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4536c800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572547.160591:0:111018:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ca80 x1875449863907072/t0(0) o36->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 600/688 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572547.160630:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863907072:192.168.202.132@tcp:36:lfs.0 00000100:00100000:2.0:1788572547.161334:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.177530:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863907072:192.168.202.132@tcp:36:lfs.0 00000080:00200000:2.0:1788572547.177581:0:111018:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e00c8) 00000080:00200000:2.0:1788572547.177587:0:111018:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x2:0x0], updating i_size 4096 00000080:00200008:2.0:1788572547.177629:0:111018:0:(file.c:479:ll_file_release()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5348), flags 300000 00000002:00100000:2.0:1788572547.177701:0:111018:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617d880 x1875449863906816/t4294983751(4294983751) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572563 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572547.177713:0:111018:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4536c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572547.177718:0:111018:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c000 x1875449863907328/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572547.177773:0:111018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863907328:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572547.177845:0:111018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.187208:0:111018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job lfs:2a90d0e1-c03b-4477-a84e-0596af2dd806:111018:1875449863907328:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572547.187251:0:111018:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617d880 x1875449863906816/t4294983751(4294983751) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572547.189075:0:110779:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:0.0:1788572547.189101:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572547.189123:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572547.189143:0:110779:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788572547.189151:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572547.189159:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:0.0:1788572547.189163:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:0.0:1788572547.189171:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e03820a7bf8 00010000:00010000:0.0:1788572547.189175:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572547.189200:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572547.189220:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908), inode mode 41ed mask 1 00000080:00200000:0.0:1788572547.189227:0:110779:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=25 00000080:00200000:0.0:1788572547.189233:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572547.189242:0:110779:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e5348), flags=25 00000080:00200000:0.0:1788572547.189246:0:110779:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e5348), intent=0 00000002:00010000:0.0:1788572547.189430:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1788572547.189483:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.189492:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.189500:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572547.189530:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907456:192.168.202.132@tcp:101:bash.0 00000100:00100000:0.0:1788572547.189617:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572547.198453:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907456:192.168.202.132@tcp:101:bash.0 00010000:00010000:0.0:1788572547.198485:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198508:0:110779:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198516:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198527:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198538:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198551:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198560:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198567:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572547.198583:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc67048380 x1875449863907456/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 592/720 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572547.198596:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198606:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589935649/2 (0 0) 00000080:00200000:0.0:1788572547.198612:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572547.198642:0:110779:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x2:0x0], updating i_size 4096 00000080:00200000:0.0:1788572547.198645:0:110779:0:(namei.c:147:ll_iget()) got inode: [0x200000421:0x2:0x0](ffff92fc665e00c8): rc = 0 00000080:00010000:0.0:1788572547.198652:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x2:0x0](ffff92fc665e00c8) for lock 0x58a604dcef352941 00000080:00010000:0.0:1788572547.198662:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x2:0x0](ffff92fc665e00c8) 00000080:00010000:0.0:1788572547.198666:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352941 from it ffff9e03820a7ac8 00010000:00010000:0.0:1788572547.198669:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572547.198689:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572547.198767:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788572547.198786:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e00c8), inode mode 41ed mask 3 00000080:00200000:0.0:1788572547.198792:0:110779:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=subf, dir=[0x200000421:0x2:0x0](ffff92fc665e00c8), file ffff92fc43419980, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1788572547.198800:0:110779:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subf, dir=[0x200000421:0x2:0x0](ffff92fc665e00c8), intent=open|creat 00000002:00010000:1.0:1788572547.281407:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 8ëUÛDAN~ý.³!±´,[0x200000421:0x3:0x0]) in obj [0x200000421:0x2:0x0], intent: open|creat flags 0101102 00010000:00010000:1.0:1788572547.281490:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 3/0,1 mode: --/CW res: [0x200000421:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572547.281502:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 3/0,1 mode: --/CW res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572547.281514:0:110779:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4536c800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572547.281526:0:110779:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6728bb80 x1875449863907712/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 808/66432 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572547.281537:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 3/0,1 mode: --/CW res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572547.281558:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907712:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572547.281710:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.294932:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907712:192.168.202.132@tcp:101:bash.0 00010000:00010000:2.0:1788572547.294979:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 4/0,1 mode: --/CW res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295003:0:110779:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 4/0,1 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295013:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 4/0,1 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295025:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 4/0,1 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295036:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 4/0,1 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295050:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 5/1,1 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295059:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 5/1,1 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295068:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572547.295644:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff92fc6728bb80 x1875449863907712/t4294983754(4294983754) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 808/696 e 0 to 0 dl 1788572563 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572547.295665:0:110779:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6728bb80 x1875449863907712/t4294983754(4294983754) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 808/696 e 0 to 0 dl 1788572563 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572547.295690:0:110779:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572547.295710:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.295938:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589935649/3 (0 0) 00000080:00200000:2.0:1788572547.295945:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000421:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572547.296008:0:110779:0:(namei.c:1628:ll_create_it()) VFS Op:name=subf, dir=[0x200000421:0x2:0x0](ffff92fc665e00c8), intent=open|creat 00000080:00200000:2.0:1788572547.296039:0:110779:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x3:0x0](ffff92fc665e6c08) 00000080:00200000:2.0:1788572547.296133:0:110779:0:(file.c:6843:ll_layout_conf()) [0x200000421:0x3:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572547.296138:0:110779:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000421:0x3:0x0](ffff92fc665e6c08) 00000080:00010000:2.0:1788572547.296141:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x3:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352948 00000080:00010000:2.0:1788572547.296230:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x3:0x0](ffff92fc665e6c08) 00000080:00200008:2.0:1788572547.296245:0:110779:0:(file.c:1090:ll_file_open()) START file subf:[0x200000421:0x3:0x0](ffff92fc665e6c08), flags 101101 00000002:00100000:2.0:1788572547.296517:0:110779:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6728bb80 x1875449863907712/t4294983754(4294983754) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572563 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572547.296552:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352948 from it ffff92fc67389a20 00010000:00010000:2.0:1788572547.296557:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.296568:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572547.296595:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x3:0x0](ffff92fc665e6c08), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572547.296905:0:110779:0:(file.c:2650:do_file_write_iter()) START file subf:[0x200000421:0x3:0x0], ppos: 0, count: 2 00000002:00010000:2.0:1788572547.296934:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200000421:0x3:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572547.297171:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 3/1,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.297189:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 3/1,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.297199:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 3/1,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572547.297223:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907840:192.168.202.132@tcp:101:bash.0 00000100:00100000:2.0:1788572547.297458:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.303544:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907840:192.168.202.132@tcp:101:bash.0 00010000:00010000:2.0:1788572547.303576:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 4/1,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.303592:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.303602:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572547.303615:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc4617c000 x1875449863907840/t0(0) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 568/1104 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572547.303626:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.303643:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc5de77800 type 13 mode 4 res 8589935649/3 (0 0) 00000080:00200000:2.0:1788572547.303651:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x200000421:0x3:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572547.303690:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x3:0x0](ffff92fc665e6c08) for lock 0x58a604dcef35294f 00000080:00010000:2.0:1788572547.303699:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef35294f from it ffff9e03820a7af0 00010000:00010000:2.0:1788572547.303704:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.303765:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572547.303804:0:110779:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 2 bytes 00000080:00200000:2.0:1788572547.303812:0:110779:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 2 00000080:00200000:2.0:1788572547.303821:0:110779:0:(file.c:2090:ll_file_io_generic()) subf: write ppos: 0, bytes: 2 00000080:00200000:2.0:1788572547.303826:0:110779:0:(file.c:1848:ll_io_set_mirror()) subf: desiginated mirror: 0 00000020:00200000:2.0:1788572547.303836:0:110779:0:(cl_io.c:214:cl_io_rw_init()) header@000000003cd83568[0x110, 1, [0x200000421:0x3:0x0] hash] 00000020:00200000:2.0:1788572547.303839:0:110779:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 2) 0 0 00000080:00200000:2.0:1788572547.303843:0:110779:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572547.303860:0:110779:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1788572547.303869:0:110779:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 2) 2, 2 00020000:00200000:2.0:1788572547.303872:0:110779:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572547.303926:0:110779:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1788572547.303940:0:110779:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572547.303945:0:110779:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550cb0, osc lock ffff92fc58370ab0, flags 0x0 00010000:00010000:2.0:1788572547.304291:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.304301:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.304311:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572547.304332:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907968:192.168.202.132@tcp:101:bash.0 00000100:00100000:2.0:1788572547.304405:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.309199:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863907968:192.168.202.132@tcp:101:bash.0 00010000:00010000:2.0:1788572547.309230:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.309250:0:110779:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.309274:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.309289:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572547.309302:0:110779:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.309313:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.309321:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572547.309351:0:110779:0:(vvp_io.c:1292:vvp_io_write_start()) subf: write [0, 2) append 0 append lockpos 0 00000080:00200000:2.0:1788572547.309378:0:110779:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 2 bytes 00000080:00200000:2.0:1788572547.309428:0:110779:0:(rw26.c:871:ll_write_end()) pos 0, len 2, copied 2 00000080:00200000:2.0:1788572547.309443:0:110779:0:(rw26.c:929:ll_write_end()) page@000000005bb194ca[3 00000000ae717f46 2 1 00000000b101f878 0 0 0] 00000080:00200000:2.0:1788572547.309447:0:110779:0:(rw26.c:929:ll_write_end()) vmpage @0000000050613879 17ffffc0001001 3:0 ffff92fc48f469a0 0 lru 00000080:00200000:2.0:1788572547.309455:0:110779:0:(rw26.c:929:ll_write_end()) osc-page@0000000006243cbb 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000ee68374a 000000007d553ef5 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1788572547.309457:0:110779:0:(rw26.c:929:ll_write_end()) end page@000000005bb194ca 00000080:00200000:2.0:1788572547.309458:0:110779:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1788572547.309462:0:110779:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 2 prio 0 00000008:00010000:2.0:1788572547.309480:0:110779:0:(osc_cache.c:858:osc_extent_find()) ### extent: 0000000034581fe5 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572547.309497:0:110779:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff92fc665e6db8, count 1, dirtied 1 00000080:00200000:2.0:1788572547.309501:0:110779:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 2 bytes, tot: 2 00000080:00200000:2.0:1788572547.309504:0:110779:0:(vvp_io.c:1358:vvp_io_write_start()) subf: write bytes 0, result: 2 00000020:00010000:2.0:1788572547.309520:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3998 (ffff92fc6721aab8/0) at cl_lock_release():245 00000020:00010000:2.0:1788572547.309523:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3998 (ffff92fc6721aab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572547.309527:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550cb0 (ffff92fc6721ab48/1) at cl_lock_cancel():142 00010000:00010000:2.0:1788572547.309532:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572547.309542:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1788572547.309557:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3998 (ffff92fc6721aab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572547.309559:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550cb0 (ffff92fc6721aab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572547.309566:0:110779:0:(vvp_io.c:533:vvp_io_advance()) advancing 2 bytes 00000080:00200000:2.0:1788572547.309573:0:110779:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1788572547.309581:0:110779:0:(vvp_io.c:304:vvp_io_fini()) [0x200000421:0x3:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572547.309584:0:110779:0:(file.c:2238:ll_file_io_generic()) subf: 2 io complete with rc: 0, result: 2, restart: 0 00000080:00200000:2.0:1788572547.309591:0:110779:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000421:0x3:0x0] updating i_size 2 i_blocks 0 00000080:00200000:2.0:1788572547.309596:0:110779:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 2 00000080:00200008:2.0:1788572547.309629:0:110779:0:(file.c:479:ll_file_release()) START file subf:[0x200000421:0x3:0x0](ffff92fc665e6c08), flags 100001 00000002:00100000:2.0:1788572547.309700:0:110779:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6728bb80 x1875449863907712/t4294983754(4294983754) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572563 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572547.310113:0:110779:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc4536c800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572547.310119:0:110779:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c000 x1875449863908224/t0(0) o35->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572547.310142:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863908224:192.168.202.132@tcp:35:bash.0 00000100:00100000:2.0:1788572547.310225:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d49000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572547.317092:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job bash:2a90d0e1-c03b-4477-a84e-0596af2dd806:110779:1875449863908224:192.168.202.132@tcp:35:bash.0 00000002:00100000:2.0:1788572547.317131:0:110779:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6728bb80 x1875449863907712/t4294983754(4294983754) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572547.461203:0:110981:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572547.696604:0:111037:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572547.696617:0:111037:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:3.0:1788572547.696819:0:111037:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572547.696840:0:111037:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572547.696857:0:111037:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572547.696865:0:111037:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572547.696872:0:111037:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:3.0:1788572547.696876:0:111037:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:3.0:1788572547.696892:0:111037:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e038231bcb8 00010000:00010000:3.0:1788572547.696896:0:111037:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572547.696905:0:111037:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572548.486657:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572549.508971:0:110981:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572549.773281:0:111045:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572549.773296:0:111045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:1.0:1788572549.773322:0:111045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572549.773342:0:111045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572549.773360:0:111045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572549.773369:0:111045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572549.773376:0:111045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:1.0:1788572549.773381:0:111045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:1.0:1788572549.773390:0:111045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0382373cb8 00010000:00010000:1.0:1788572549.773394:0:111045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572549.773405:0:111045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200008:1.0:1788572549.773558:0:111045:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665e0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572549.773563:0:111045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908),name=/ 00000002:00010000:1.0:1788572549.773571:0:111045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572549.773578:0:111045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572549.773586:0:111045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572549.773594:0:111045:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572549.773598:0:111045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00010000:1.0:1788572549.773601:0:111045:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665e0908) for lock 0x58a604dcef352917 00000080:00010000:1.0:1788572549.773604:0:111045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352917 from it ffff9e0382373cb8 00010000:00010000:1.0:1788572549.773607:0:111045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572549.773614:0:111045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572549.816363:0:31488:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572549.816385:0:31488:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1788572549.816415:0:31488:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 0000000034581fe5 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788572549.816434:0:31488:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 0000000034581fe5 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572549.816492:0:31488:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00100000:2.0:1788572549.816620:0:31488:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617d880 - object 0x2c0000401:538 offset 0<>4096 00000080:00200000:2.0:1788572549.816682:0:31488:0:(vvp_io.c:304:vvp_io_fini()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788572549.816782:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2a90d0e1-c03b-4477-a84e-0596af2dd806:2396:1875449863908352:192.168.202.132@tcp:4:bash.0 00000008:00010000:2.0:1788572549.847153:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 0000000034581fe5 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00010000:2.0:1788572549.847245:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 0000000034581fe5 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00100000:2.0:1788572549.847274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:2a90d0e1-c03b-4477-a84e-0596af2dd806:2396:1875449863908352:192.168.202.132@tcp:4:bash.0 00000080:00200000:1.0:1788572549.847503:0:111045:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665e0908) 00000080:00200000:1.0:1788572549.847545:0:111045:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5348) 00000080:00200000:1.0:1788572549.847565:0:111045:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e00c8) 00000080:00200000:1.0:1788572549.847617:0:111045:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572549.847630:0:111045:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572549.847684:0:111045:0:(vvp_io.c:304:vvp_io_fini()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572549.847795:0:111045:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x3:0x0](ffff92fc665e6c08) 00000008:00010000:1.0:1788572549.847861:0:111045:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 2 blocks 8 [cma]time 1788572547 1788572547 0 ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 1/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572549.847949:0:111045:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc4536c800 (sb ffff92fc4536c800) 00010000:00010000:1.0:1788572549.847995:0:111045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848004:0:111045:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848016:0:111045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848022:0:111045:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848042:0:111045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572549.848047:0:111045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848054:0:111045:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848061:0:111045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848068:0:111045:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848079:0:111045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1788572549.848085:0:111045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848091:0:111045:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848097:0:111045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848108:0:111045:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.848116:0:111045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:1.0:1788572549.848119:0:111045:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:1.0:1788572549.848629:0:111045:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc5824f800 active: 1 00020000:01000000:1.0:1788572549.848634:0:111045:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc5824d800 active: 1 00020000:01000000:1.0:1788572549.848637:0:111045:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:1.0:1788572549.848641:0:111045:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc4536c800 00020000:01000000:1.0:1788572549.848643:0:111045:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc4536c800: disconnecting target lustre-OST0001-osc-ffff92fc4536c800 00000100:00080000:1.0:1788572549.848654:0:111045:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-OST0001_UUID 00010000:00010000:1.0:1788572549.848691:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572549.848773:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 1/0,0 mode: --/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572549.848782:0:111045:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572549.848797:0:111045:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788572549.848806:0:111045:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job umount:2a90d0e1-c03b-4477-a84e-0596af2dd806:111045:1875449863908480:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572549.848813:0:111045:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572549.848821:0:111045:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572549.848824:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff92fc4536c800 lock: 000000005542eb9e/0x58a604dcef352956 lrc: 0/0,0 mode: --/PW res: [0x2c0000401:0x21a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792c39 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000100:00080000:1.0:1788572549.848905:0:111045:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:0.0:1788572549.849158:0:110980:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1788572549.849165:0:110980:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849188:0:110980:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205600 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572549.849192:0:110980:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849201:0:110980:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572549.849207:0:110980:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572549.849213:0:110980:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572549.849215:0:110980:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:0.0:1788572549.849218:0:110980:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:0.0:1788572549.849239:0:110980:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849245:0:110980:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572549.849270:0:110980:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849277:0:110980:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70205600/0x58a604dcef3528a7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b13 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849287:0:110980:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788572549.849290:0:110980:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849296:0:110980:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204600 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572549.849299:0:110980:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849306:0:110980:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572549.849313:0:110980:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572549.849318:0:110980:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572549.849320:0:110980:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:0.0:1788572549.849321:0:110980:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:0.0:1788572549.849337:0:110980:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849342:0:110980:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572549.849348:0:110980:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849354:0:110980:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70204600/0x58a604dcef3528d8 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b1a expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849366:0:110980:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572549.849370:0:110980:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849375:0:110980:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204400 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572549.849378:0:110980:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849384:0:110980:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572549.849394:0:110980:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572549.849399:0:110980:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:0.0:1788572549.849400:0:110980:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:0.0:1788572549.849402:0:110980:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:0.0:1788572549.849414:0:110980:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849418:0:110980:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572549.849422:0:110980:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849427:0:110980:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70204400/0x58a604dcef3528df lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b21 expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572549.849436:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572549.849438:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572549.849440:0:110980:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572549.849456:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863908608:192.168.202.132@tcp:9:umount.0 00000100:00100000:0.0:1788572549.849562:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4761e684-853a-44f0-be14-fa356e387a10:2394:1875449863908864:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572549.849594:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788572549.849601:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4761e684-853a-44f0-be14-fa356e387a10:2394:1875449863908736:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572549.849621:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4761e684-853a-44f0-be14-fa356e387a10:2394:1875449863908992:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572549.849871:0:110981:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572549.852781:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572549.852808:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863908608:192.168.202.132@tcp:9:umount.0 00000100:00100000:0.0:1788572549.852895:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4761e684-853a-44f0-be14-fa356e387a10:2394:1875449863908736:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572549.852906:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4761e684-853a-44f0-be14-fa356e387a10:2394:1875449863908992:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572549.852913:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4761e684-853a-44f0-be14-fa356e387a10:2394:1875449863908864:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:1.0:1788572549.853265:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572549.853269:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4536c800: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572549.853280:0:111045:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617d880 x1875449863908352/t4294968474(4294968474) o4->lustre-OST0001-osc-ffff92fc4536c800@192.168.202.132@tcp:6/4 lens 4584/448 e 0 to 0 dl 1788572565 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572549.853299:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572549.853424:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352909 00000020:00000080:1.0:1788572549.853440:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc5824d800/LOV_OSC_UUID 00000020:01000000:1.0:1788572549.853494:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc4536c800 (flags='') 00000020:00000080:1.0:1788572549.853501:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572549.853510:0:111045:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc4536c800: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572549.853514:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc49930000) has no exports 00000020:00000080:2.0:1788572549.853832:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824d800/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc4536c800 00000008:01000000:0.0:1788572549.854320:0:111045:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc4536c800: client import never connected 00000100:00080000:0.0:1788572549.854325:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572549.854327:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc4536c800: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572549.854331:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572549.854523:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572549.854530:0:111045:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00020000:01000000:0.0:1788572549.854534:0:111045:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc4536c800 00020000:01000000:0.0:1788572549.854536:0:111045:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc4536c800: disconnecting target lustre-OST0000-osc-ffff92fc4536c800 00000100:00080000:0.0:1788572549.854547:0:111045:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-OST0000_UUID 00000100:00080000:0.0:1788572549.854646:0:111045:0:(import.c:65:import_set_state_nolock()) ffff92fc51069800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572549.854754:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863909120:192.168.202.132@tcp:9:umount.0 00000020:00000080:2.0:1788572549.855148:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51d2d000 for lustre-OST0001-osc-ffff92fc4536c800 00000020:00000080:2.0:1788572549.855157:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc51d2c800/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:2.0:1788572549.855160:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2c800/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-OST0001-osc-ffff92fc4536c800 00000020:01000000:2.0:1788572549.855166:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc4536c800 (2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000100:00080000:3.0:1788572549.860704:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51069800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572549.860775:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863909120:192.168.202.132@tcp:9:umount.0 00000100:00080000:0.0:1788572549.860900:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572549.860905:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4536c800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572549.860910:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572549.861051:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352902 00000020:00000080:0.0:1788572549.861071:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc5824f800/LOV_OSC_UUID 00000020:01000000:0.0:1788572549.861115:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc4536c800 (flags='') 00000020:00000080:0.0:1788572549.861120:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572549.861128:0:111045:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc4536c800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572549.861132:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc499319c0) has no exports 00000020:00000080:2.0:1788572549.861494:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824f800/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc4536c800 00000008:01000000:0.0:1788572549.862866:0:111045:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc4536c800: client import never connected 00000100:00080000:0.0:1788572549.862874:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572549.862878:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc4536c800: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572549.862884:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572549.863086:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572549.863095:0:111045:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:00000080:0.0:1788572549.863100:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3528fb 00000020:00000080:0.0:1788572549.863108:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc58248000/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000100:00080000:0.0:1788572549.863190:0:111045:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-MDT0000_UUID 00010000:00010000:0.0:1788572549.863232:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572549.863282:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 1/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572549.863290:0:111045:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572549.863306:0:111045:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 0 seconds 00000100:00100000:0.0:1788572549.863318:0:111045:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job umount:2a90d0e1-c03b-4477-a84e-0596af2dd806:111045:1875449863909248:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572549.863327:0:111045:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572549.863335:0:111045:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572549.863339:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f4e00/0x58a604dcef35291e lrc: 0/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792b8a expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572549.863374:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863391:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863397:0:111045:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572549.863402:0:111045:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 0 seconds 00000100:00100000:0.0:1788572549.863407:0:111045:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job umount:2a90d0e1-c03b-4477-a84e-0596af2dd806:111045:1875449863909376:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572549.863412:0:111045:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572549.863415:0:111045:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572549.863418:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc3c00/0x58a604dcef352941 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792c01 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863452:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863463:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863477:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 1/0,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863484:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863489:0:111045:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:0.0:1788572549.863494:0:111045:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 0 seconds 00000100:00100000:0.0:1788572549.863497:0:111045:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job umount:2a90d0e1-c03b-4477-a84e-0596af2dd806:111045:1875449863909504:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572549.863502:0:111045:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572549.863506:0:111045:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572549.863508:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc70367800/0x58a604dcef35294f lrc: 0/0,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792c32 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863516:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc702f5e00/0x58a604dcef352948 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792c1d expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572549.863583:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572549.863599:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788572549.863606:0:111045:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572549.863612:0:111045:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 0 seconds 00000100:00100000:0.0:1788572549.863616:0:111045:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job umount:2a90d0e1-c03b-4477-a84e-0596af2dd806:111045:1875449863909632:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572549.863621:0:111045:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572549.863625:0:111045:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572549.863627:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc4536c800 lock: ffff92fc51bc2800/0x58a604dcef352917 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792b6e expref: -99 pid: 111018 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000020:00000080:2.0:1788572549.863828:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51069800 for lustre-OST0000-osc-ffff92fc4536c800 00000020:00000080:2.0:1788572549.863836:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc5106e000/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:2.0:1788572549.863839:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5106e000/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-OST0000-osc-ffff92fc4536c800 00000020:01000000:2.0:1788572549.863845:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc4536c800 (2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:00000080:2.0:1788572549.864091:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58248000/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-clilov-ffff92fc4536c800 00000100:00080000:0.0:1788572549.866972:0:111045:0:(import.c:65:import_set_state_nolock()) ffff92fc5838d800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572549.869062:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863909760:192.168.202.132@tcp:39:umount.0 00000100:00080000:3.0:1788572549.875036:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5838d800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572549.875067:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863909760:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572549.875178:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572549.875182:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4536c800: committing for last_committed 4294983751 gen 2 00000100:00100000:0.0:1788572549.875193:0:111045:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617ca80 x1875449863907072/t4294983752(4294983752) o36->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 600/448 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572549.875217:0:111045:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6728bb80 x1875449863907712/t4294983754(4294983754) o101->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572549.875231:0:111045:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617c000 x1875449863908224/t4294983755(4294983755) o35->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572549.875241:0:111045:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67048000 x1875449863909760/t4294983756(4294983756) o39->lustre-MDT0000-mdc-ffff92fc4536c800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572559 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572549.875253:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572549.877062:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3528ed 00000020:00000080:0.0:1788572549.877077:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc58248800/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000100:00080000:0.0:1788572549.877180:0:111045:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 2a90d0e1-c03b-4477-a84e-0596af2dd806->lustre-MDT0001_UUID 00000020:00000080:1.0:1788572549.878121:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58248800/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-MDT0000-mdc-ffff92fc4536c800 00000100:00080000:0.0:1788572549.885276:0:111045:0:(import.c:65:import_set_state_nolock()) ffff92fc5106b000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572549.885394:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863909888:192.168.202.132@tcp:39:umount.0 00000100:00100000:3.0:1788572549.890377:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4536c800: committing for last_committed 4294976752 gen 1 00000100:00080000:3.0:1788572549.890393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5106b000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572549.890423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2a90d0e1-c03b-4477-a84e-0596af2dd806:2392:1875449863909888:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572549.891012:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572549.891018:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4536c800: committing for last_committed 4294976752 gen 2 00000100:00100000:0.0:1788572549.891024:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572549.892862:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3528f4 00000020:00000080:0.0:1788572549.892875:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc5824f000/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:0.0:1788572549.892910:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3528e6 00000020:00000080:0.0:1788572549.892914:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc5824e800/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:01000000:0.0:1788572549.892921:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc4536c800 (flags='') 00000020:00000080:0.0:1788572549.892930:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572549.892937:0:111045:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc4536c800: forcing exports to disconnect: 0/1 00000020:00080000:0.0:1788572549.892941:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc70033380) has no exports 00000020:00000080:0.0:1788572549.893000:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572549.893005:0:111045:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:00000080:0.0:1788572549.893008:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc5824e000/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:0.0:1788572549.893010:0:111045:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824e000/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-clilov-ffff92fc4536c800 00000020:01000000:0.0:1788572549.893015:0:111045:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc4536c800 (2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:0.0:1788572549.893160:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc4536c800 (flags='') 00000020:00000080:0.0:1788572549.893164:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572549.893178:0:111045:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc4536c800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572549.893181:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc700319c0) has no exports 00000020:00000080:0.0:1788572549.893219:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572549.893224:0:111045:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:0.0:1788572549.893227:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc4536c800 (flags='') 00000020:00000080:0.0:1788572549.893230:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572549.893232:0:111045:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc4536c800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572549.893234:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc49933380) has no exports 00000020:00000080:1.0:1788572549.893523:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824f000/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-MDT0001-mdc-ffff92fc4536c800 00000020:00000080:1.0:1788572549.893532:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5824e800/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-clilmv-ffff92fc4536c800 00000020:00000080:1.0:1788572549.893536:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc58249000/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:1.0:1788572549.893538:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58249000/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-clilmv-ffff92fc4536c800 00000020:01000000:1.0:1788572549.893542:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc4536c800 (2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000008:01000000:0.0:1788572549.900295:0:111045:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc4536c800: client import never connected 00000100:00080000:0.0:1788572549.900306:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572549.900311:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc4536c800: committing for last_committed 4294983751 gen 3 00000100:00100000:0.0:1788572549.900321:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572549.912317:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572549.912330:0:111045:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000020:01000000:1.0:1788572549.912336:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc4536c800 (flags='') 00000020:00000080:1.0:1788572549.912340:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572549.912343:0:111045:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc4536c800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572549.912348:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc49934d40) has no exports 00000020:00000080:3.0:1788572549.912761:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5838d800 for lustre-MDT0000-mdc-ffff92fc4536c800 00000020:00000080:3.0:1788572549.912771:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc58388800/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:3.0:1788572549.912774:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58388800/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-MDT0000-mdc-ffff92fc4536c800 00000020:01000000:3.0:1788572549.912784:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc4536c800 (2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000008:01000000:0.0:1788572549.915035:0:111045:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc4536c800: client import never connected 00000100:00080000:0.0:1788572549.915042:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572549.915047:0:111045:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc4536c800: committing for last_committed 4294976752 gen 3 00000100:00100000:0.0:1788572549.915059:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572549.926846:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572549.926861:0:111045:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc4536c800 (uuid 2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000080:02000400:1.0:1788572549.926866:0:111045:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572549.926869:0:111045:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1788572549.926926:0:111045:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc4536c800 00000100:00080000:1.0:1788572549.926935:0:111045:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 4761e684-853a-44f0-be14-fa356e387a10->MGS 00010000:00010000:1.0:1788572549.926950:0:111045:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792b0c expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572549.926964:0:111045:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792b0c expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572549.926969:0:111045:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572549.926972:0:111045:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572549.926974:0:111045:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572549.927038:0:111045:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b0c expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572549.927045:0:111045:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572549.927062:0:111045:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58322e40 going to sleep for 0 seconds 00000100:00100000:1.0:1788572549.927074:0:111045:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job umount:4761e684-853a-44f0-be14-fa356e387a10:111045:1875449863910016:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572549.927086:0:111045:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572549.927094:0:111045:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572549.927097:0:111045:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2c00/0x58a604dcef3528a0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792b0c expref: -99 pid: 110969 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572549.927128:0:111045:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0f800 MGS: changing import state from FULL to CONNECTING 00000020:00000080:0.0:1788572549.932053:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc5106b000 for lustre-MDT0001-mdc-ffff92fc4536c800 00000020:00000080:0.0:1788572549.932071:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc5106f000/2a90d0e1-c03b-4477-a84e-0596af2dd806 00000020:00000080:0.0:1788572549.932074:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5106f000/2a90d0e1-c03b-4477-a84e-0596af2dd806 for lustre-MDT0001-mdc-ffff92fc4536c800 00000020:01000000:0.0:1788572549.932089:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc4536c800 (2a90d0e1-c03b-4477-a84e-0596af2dd806) 00000100:00100000:3.0:1788572549.944556:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4761e684-853a-44f0-be14-fa356e387a10:2392:1875449863910144:192.168.202.132@tcp:251:umount.0 00000100:00080000:2.0:1788572549.962524:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0f800 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:2.0:1788572549.962535:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572549.962539:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572549.962540:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572549.962559:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4761e684-853a-44f0-be14-fa356e387a10:2392:1875449863910144:192.168.202.132@tcp:251:umount.0 00000100:00080000:1.0:1788572549.962810:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572549.962813:0:111045:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572549.962816:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572549.962818:0:111045:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572549.962829:0:111045:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352899 00000020:00000080:1.0:1788572549.962838:0:111045:0:(genops.c:885:class_export_put()) final put ffff92fc492a7800/4761e684-853a-44f0-be14-fa356e387a10 00000020:01000000:1.0:1788572549.962851:0:111045:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:1.0:1788572549.962857:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572549.962864:0:111045:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572549.962868:0:111045:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc70034d40) has no exports 00000020:00000080:1.0:1788572549.962906:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a7800/4761e684-853a-44f0-be14-fa356e387a10 for MGC192.168.202.132@tcp 10000000:01000000:1.0:1788572549.965075:0:110983:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788572549.965254:0:111045:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:2.0:1788572549.965259:0:111045:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572549.965262:0:111045:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572549.965267:0:111045:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572549.965269:0:111045:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572549.965342:0:111045:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572549.965350:0:111045:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 4761e684-853a-44f0-be14-fa356e387a10) 00000020:01000004:2.0:1788572549.965355:0:111045:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc4536c800 1 00000020:01000004:2.0:1788572549.965359:0:111045:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc492a0000 00000020:00000080:1.0:1788572549.965497:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc42c0f800 for MGC192.168.202.132@tcp 00000020:00000080:1.0:1788572549.965505:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc492a3800/4761e684-853a-44f0-be14-fa356e387a10 00000020:00000080:1.0:1788572549.965509:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a3800/4761e684-853a-44f0-be14-fa356e387a10 for MGC192.168.202.132@tcp 00000020:01000000:1.0:1788572549.965515:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (4761e684-853a-44f0-be14-fa356e387a10) 00010000:00010000:1.0:1788572549.965811:0:110980:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572549.965922:0:110981:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00000100:00100000:1.0:1788572549.966059:0:110971:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc49d30600:110971] 1 exiting: rc = 0 00000100:00100000:0.0:1788572549.968636:0:110970:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc46538e00:110970] 0 exiting: rc = 1 00000100:00100000:2.0:1788572549.977586:0:110973:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff92fc49d71000:110973] 1 exiting: rc = 0 00000100:00100000:3.0:1788572549.978142:0:110972:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc49d70600:110972] 0 exiting: rc = 1 00000100:00100000:1.0:1788572549.978365:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572549.978375:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572549.978590:0:110975:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc4328ba00:110975] 1 exiting: rc = 0 00000100:00100000:0.0:1788572549.984521:0:110974:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc491e9200:110974] 0 exiting: rc = 1 00000100:00100000:0.0:1788572549.984596:0:110976:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc470f3800:110976] 2 exiting: rc = 0 00000100:00100000:2.0:1788572550.010300:0:110977:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc4328a800:110977] 0 exiting: rc = 1 00000100:00100000:2.0:1788572550.010399:0:110978:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc470f2c00:110978] 1 exiting: rc = 0 00000100:00100000:2.0:1788572550.010449:0:110979:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc470f3e00:110979] 2 exiting: rc = 0 00000100:00100000:0.0:1788572550.028125:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028136:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028142:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028147:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028153:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028159:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028165:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028184:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028189:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028195:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028200:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028205:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028210:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028215:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028221:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028226:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028232:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028236:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028242:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028248:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028255:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028261:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028266:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028271:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028277:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028282:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028288:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028293:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028300:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028305:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028311:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028316:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028322:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028328:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028334:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028339:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028345:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028350:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028356:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028361:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028366:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028372:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028378:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028383:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028388:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028394:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028400:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028406:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028412:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028418:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028424:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028429:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028434:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028439:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028445:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028450:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028456:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028461:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028467:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028472:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028477:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028482:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028488:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028494:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028502:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028508:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028513:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028519:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028524:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028529:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028535:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028540:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028545:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028550:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028555:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028560:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028565:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028571:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028576:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028582:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028588:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028593:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028599:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028604:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028609:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028614:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028620:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028624:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028630:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028635:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028640:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028645:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028650:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028654:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028660:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028665:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028671:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028676:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028682:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028687:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028693:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028698:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028703:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028709:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028715:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028769:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028778:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028784:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028789:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028794:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028799:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028805:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028810:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028815:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028821:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028826:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028831:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028837:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028843:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028848:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028853:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028859:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028865:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028871:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028877:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028882:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028888:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572550.028893:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:1.0:1788572550.248303:0:111058:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:1.0:1788572550.248308:0:111058:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:1.0:1788572550.248309:0:111058:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:1.0:1788572550.248310:0:111058:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:1.0:1788572550.248414:0:111058:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc51d2a800 00000080:01000004:1.0:1788572550.248424:0:111058:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:1.0:1788572550.248440:0:111058:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:1.0:1788572550.248466:0:111058:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:1.0:1788572550.248479:0:111058:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc49934d40) 00000020:00000080:1.0:1788572550.248490:0:111058:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:1.0:1788572550.248644:0:111058:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572550.248676:0:111058:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:0.0:1788572550.249619:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:0.0:1788572550.249628:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572550.253486:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572550.253496:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572550.255611:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:2.0:1788572550.255628:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:0.0:1788572550.257767:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572550.257779:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:2.0:1788572550.261101:0:111058:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:2.0:1788572550.261397:0:111058:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:2.0:1788572550.261575:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:2.0:1788572550.261582:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788572550.264571:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788572550.264649:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:0.0:1788572550.265859:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:0.0:1788572550.265963:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788572550.270987:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788572550.271012:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:0.0:1788572550.272358:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:0.0:1788572550.272367:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:1.0:1788572550.278051:0:111058:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:1.0:1788572550.278067:0:111058:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:1.0:1788572550.286574:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572550.286580:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572550.286583:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572550.287127:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572550.287131:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572550.287132:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:0.0:1788572550.295316:0:111058:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc42c08000@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572550.295479:0:111058:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:0.0:1788572550.295486:0:111058:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:0.0:1788572550.297480:0:111072:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:0.0:1788572550.297517:0:111058:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid 71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1) 00000020:00000080:0.0:1788572550.297540:0:111058:0:(genops.c:1385:class_connect()) connect: client 71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1, cookie 0x58a604dcef352964 00000100:00080000:0.0:1788572550.297555:0:111058:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572550.297566:0:111058:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc42c08000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572550.297608:0:111058:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449863910272/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572550.297633:0:111058:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1->MGS 00000080:00200000:0.0:1788572550.297652:0:111058:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc51d2a800 (sb ffff92fc51d2a800) 00000080:01000000:0.0:1788572550.297715:0:111058:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:0.0:1788572550.297718:0:111058:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788572550.297739:0:111058:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:0.0:1788572550.297748:0:111058:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 969e09c0-9457-45fe-bdd1-1282a4dee8fd 10000000:01000000:0.0:1788572550.298163:0:111058:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788572550.298178:0:111058:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc51d2a800 10000000:01000000:0.0:1788572550.298182:0:111058:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc49934d40 10000000:01000000:0.0:1788572550.298186:0:111058:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572550.298194:0:111058:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc49934d40 from 1 10000000:01000000:0.0:1788572550.298197:0:111058:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572550.298227:0:111058:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572550.298238:0:111058:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572550.298250:0:111058:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572550.298270:0:111058:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67048700 x1875449863910400/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572550.298282:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc43af99c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788572550.304320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2392:1875449863910272:192.168.202.132@tcp:250:mount.lustre.0 00010000:00010000:2.0:1788572550.304750:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572550.315503:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572550.315517:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:3.0:1788572550.315525:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:3.0:1788572550.315531:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:3.0:1788572550.315536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c08000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:3.0:1788572550.315539:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572550.315553:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:3.0:1788572550.315565:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000083d843e0): req@ffff92fc67048700 x1875449863910400/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572550.315586:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2392:1875449863910272:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:1.0:1788572550.315752:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc43af99c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572550.319662:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910400:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572550.319694:0:111058:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572550.319708:0:111058:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572550.319823:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910528:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572550.319889:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45187480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.332750:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910528:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:3.0:1788572550.332999:0:111058:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:3.0:1788572550.333021:0:111058:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572550.333032:0:111058:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572550.333041:0:111058:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc51d2a800 10000000:01000000:3.0:1788572550.333048:0:111058:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:3.0:1788572550.333053:0:111058:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc51d2a800 10000000:01000000:3.0:1788572550.333056:0:111058:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:3.0:1788572550.333060:0:111058:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc51d2a800 10000000:01000000:3.0:1788572550.333062:0:111058:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:3.0:1788572550.333067:0:111058:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc51d2a800 from 1 10000000:01000000:3.0:1788572550.333069:0:111058:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:3.0:1788572550.333108:0:111058:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572550.333116:0:111058:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572550.333128:0:111058:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572550.333152:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910656:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572550.333255:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc609880c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572550.342279:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910656:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572550.342308:0:111058:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572550.342323:0:111058:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572550.342410:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910784:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572550.342505:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc47154a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572550.345466:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910784:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572550.345532:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910912:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572550.345667:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc47154a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572550.350541:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863910912:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572550.351459:0:111076:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111076:1875449863911040:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572550.351546:0:111076:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc47154000 going to sleep for 11 seconds 00000100:00100000:0.0:1788572550.372387:0:111076:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111076:1875449863911040:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572550.372437:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.372449:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.372451:0:111076:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:0.0:1788572550.372459:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.372462:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572550.372475:0:111076:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc51d2a800 (ffff92fc70034d40) 00000020:00000080:0.0:1788572550.372498:0:111076:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:0.0:1788572550.372505:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.372507:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572550.372832:0:111076:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572550.372847:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.372851:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.372853:0:111076:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:0.0:1788572550.372855:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.372859:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.372860:0:111076:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572550.372864:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.372866:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572550.372872:0:111076:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc51d2a800 (ffff92fc70033380) 00000020:00000080:0.0:1788572550.372881:0:111076:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:0.0:1788572550.375063:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.375080:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572550.375351:0:111076:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572550.375361:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.375366:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.375368:0:111076:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572550.375371:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.375375:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.375377:0:111076:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:0.0:1788572550.375380:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:0.0:1788572550.375382:0:111076:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:0.0:1788572550.375385:0:111076:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:0.0:1788572550.375390:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.375392:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.375393:0:111076:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:0.0:1788572550.375395:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.375399:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.375400:0:111076:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:0.0:1788572550.375402:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572550.375407:0:111076:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572550.375429:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.375431:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572550.375441:0:111076:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc51d2a800 (ffff92fc70030000) 00000020:00000080:0.0:1788572550.375455:0:111076:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:0.0:1788572550.375460:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.375462:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572550.375518:0:111076:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc46304800@lustre-MDT0000-mdc-ffff92fc51d2a800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572550.389246:0:111076:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc51d2a800-1 is initialized 00000040:01000000:0.0:1788572550.389484:0:111076:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc51d2a800 ctxt 13 is initialized 00000020:00000080:0.0:1788572550.389649:0:111076:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572550.389664:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.389668:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572550.389694:0:111076:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:0.0:1788572550.389748:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.389752:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.389754:0:111076:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:0.0:1788572550.389757:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572550.389759:0:111076:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:0.0:1788572550.389761:0:111076:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572550.389762:0:111076:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572550.389763:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572550.389766:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.389767:0:111076:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572550.389770:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.389774:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.389775:0:111076:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:0.0:1788572550.389777:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572550.389782:0:111076:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572550.393630:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.393635:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572550.393648:0:111076:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc51d2a800 (ffff92fc700319c0) 00000020:00000080:0.0:1788572550.393667:0:111076:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:0.0:1788572550.393672:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.393674:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572550.394040:0:111076:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4536c800@lustre-MDT0001-mdc-ffff92fc51d2a800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572550.402226:0:111076:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc51d2a800-2 is initialized 00000040:01000000:0.0:1788572550.402456:0:111076:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc51d2a800 ctxt 13 is initialized 00000020:00000080:0.0:1788572550.404549:0:111076:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572550.404565:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.404570:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572550.404597:0:111076:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:0.0:1788572550.404813:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.404817:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.404819:0:111076:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:0.0:1788572550.404822:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572550.404825:0:111076:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:0.0:1788572550.404827:0:111076:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572550.404829:0:111076:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572550.404831:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572550.404833:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.404834:0:111076:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572550.404836:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572550.404842:0:111076:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:0.0:1788572550.404846:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.404847:0:111076:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:0.0:1788572550.404850:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572550.404855:0:111076:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572550.404882:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.404885:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572550.404977:0:111076:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc51d2a800 (ffff92fc51cd99c0) 00000020:00000080:0.0:1788572550.404994:0:111076:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:0.0:1788572550.405000:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.405003:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572550.405094:0:111076:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc492a3800@lustre-OST0000-osc-ffff92fc51d2a800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572550.405329:0:111076:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc51d2a800-3 is initialized 00000020:00000080:0.0:1788572550.405545:0:111076:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572550.405552:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.405555:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572550.405568:0:111076:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:0.0:1788572550.405574:0:111076:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc51d2a800: Adding target lustre-OST0000_UUID 00020000:01000000:0.0:1788572550.405579:0:111076:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:0.0:1788572550.405589:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.405592:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.405593:0:111076:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:0.0:1788572550.405596:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572550.405598:0:111076:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:0.0:1788572550.405599:0:111076:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572550.405601:0:111076:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572550.405603:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572550.405605:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.405606:0:111076:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572550.405609:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572550.405612:0:111076:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:0.0:1788572550.405615:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.405616:0:111076:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:0.0:1788572550.405619:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572550.405622:0:111076:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572550.405635:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.405638:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572550.405644:0:111076:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc51d2a800 (ffff92fc51cdb380) 00000020:00000080:0.0:1788572550.405654:0:111076:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:0.0:1788572550.405659:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.405661:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572550.405704:0:111076:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc492a5000@lustre-OST0001-osc-ffff92fc51d2a800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572550.407533:0:111076:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc51d2a800-4 is initialized 00000020:00000080:0.0:1788572550.410583:0:111076:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572550.410605:0:111076:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc51d2a800 00000020:00000080:0.0:1788572550.410611:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572550.410636:0:111076:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:0.0:1788572550.410645:0:111076:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc51d2a800: Adding target lustre-OST0001_UUID 00020000:01000000:0.0:1788572550.410650:0:111076:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:0.0:1788572550.410659:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.410662:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.410664:0:111076:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:0.0:1788572550.410668:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.410671:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.410673:0:111076:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:0.0:1788572550.410675:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:0.0:1788572550.410677:0:111076:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:0.0:1788572550.410679:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572550.410682:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.410683:0:111076:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:0.0:1788572550.410748:0:111076:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111076:1875449863911168:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572550.414005:0:111076:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a240 going to sleep for 11 seconds 00000100:00100000:0.0:1788572550.424380:0:111076:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111076:1875449863911168:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572550.424436:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572550.424451:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572550.424454:0:111076:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:0.0:1788572550.424460:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:0.0:1788572550.424503:0:111076:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111076:1875449863911296:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572550.424613:0:111076:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572550.431430:0:111076:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111076:1875449863911296:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572550.431487:0:111076:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572550.431496:0:111076:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572550.431498:0:111076:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:2.0:1788572550.431504:0:111076:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:3.0:1788572550.431629:0:111058:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:3.0:1788572550.431649:0:111058:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572550.431676:0:111058:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572550.431688:0:111058:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572550.431696:0:111058:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc51d2a800 from 1 10000000:01000000:3.0:1788572550.431700:0:111058:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572550.431855:0:111058:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.431865:0:111058:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.431875:0:111058:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572550.431964:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911424:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572550.432130:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fec00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572550.434819:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911424:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572550.434847:0:111058:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.434860:0:111058:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572550.438978:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911552:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572550.439083:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fec00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572550.453139:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911552:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572550.454086:0:111058:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572550.454187:0:111058:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.454200:0:111058:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572550.454209:0:111058:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc51d2a800 from 1 10000000:01000000:2.0:1788572550.454211:0:111058:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572550.454250:0:111058:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.454258:0:111058:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.454269:0:111058:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572550.454292:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911680:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572550.454377:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fec00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572550.464101:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911680:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572550.464133:0:111058:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572550.464148:0:111058:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572550.464593:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911808:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572550.464844:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.467068:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911808:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572550.467131:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911936:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572550.467232:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.473357:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111058:1875449863911936:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572550.479067:0:111081:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111081:1875449863912064:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572550.479215:0:111081:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988240 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.482366:0:111081:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111081:1875449863912064:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572550.482415:0:111081:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572550.482426:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572550.482429:0:111081:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:3.0:1788572550.482436:0:111081:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc51d2a800 00000020:00000080:3.0:1788572550.482438:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572550.482447:0:111081:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572550.482449:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572550.482451:0:111081:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:3.0:1788572550.482453:0:111081:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572550.482455:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572550.482456:0:111081:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:3.0:1788572550.482459:0:111081:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc51d2a800 00000020:00000080:3.0:1788572550.482461:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572550.482517:0:111081:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572550.482521:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572550.482522:0:111081:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:3.0:1788572550.482525:0:111081:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572550.482528:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572550.482529:0:111081:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:3.0:1788572550.482533:0:111081:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc51d2a800 00000020:00000080:3.0:1788572550.482535:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:0.0:1788572550.485592:0:111081:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 3050us 00000100:00100000:0.0:1788572550.485653:0:111081:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111081:1875449863912192:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572550.486794:0:111081:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.494058:0:111081:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job llog_process_th:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111081:1875449863912192:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572550.494112:0:111081:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572550.494122:0:111081:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572550.494124:0:111081:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:3.0:1788572550.494128:0:111081:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:0.0:1788572550.503076:0:111058:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:0.0:1788572550.503107:0:111058:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788572550.503122:0:111058:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572550.503135:0:111058:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:0.0:1788572550.503155:0:111058:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572550.503194:0:111058:0:(genops.c:1385:class_connect()) connect: client 969e09c0-9457-45fe-bdd1-1282a4dee8fd, cookie 0x58a604dcef3529b1 00800000:01000000:0.0:1788572550.503219:0:111058:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 969e09c0-9457-45fe-bdd1-1282a4dee8fd to lustre-clilmv-ffff92fc51d2a800 00800000:01000000:0.0:1788572550.503228:0:111058:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc51d2a800(969e09c0-9457-45fe-bdd1-1282a4dee8fd) - lustre-MDT0000_UUID, 969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:0.0:1788572550.503236:0:111058:0:(genops.c:1385:class_connect()) connect: client 969e09c0-9457-45fe-bdd1-1282a4dee8fd, cookie 0x58a604dcef3529b8 00000100:00080000:0.0:1788572550.503250:0:111058:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc51d2a800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572550.503262:0:111058:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc51d2a800: import ffff92fc46304800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572550.503279:0:111058:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572550.503318:0:111058:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449863912320/t0(0) o38->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572550.503406:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863912320:192.168.202.132@tcp:38:mount.lustre.0 00000100:00080000:0.0:1788572550.504310:0:111058:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-MDT0000_UUID 00800000:01000000:0.0:1788572550.504421:0:111058:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc51d2a800(969e09c0-9457-45fe-bdd1-1282a4dee8fd) successfully (5) 00800000:00000004:0.0:1788572550.504426:0:111058:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc51d2a800: age 3306, max_age 0 00800000:01000000:0.0:1788572550.504470:0:111058:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc51d2a800(969e09c0-9457-45fe-bdd1-1282a4dee8fd) - lustre-MDT0001_UUID, 969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:0.0:1788572550.504482:0:111058:0:(genops.c:1385:class_connect()) connect: client 969e09c0-9457-45fe-bdd1-1282a4dee8fd, cookie 0x58a604dcef3529bf 00000100:00080000:0.0:1788572550.504489:0:111058:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc51d2a800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572550.504497:0:111058:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc51d2a800: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572550.504505:0:111058:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572550.504533:0:111058:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863912576/t0(0) o38->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572550.504614:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1788572550.504629:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67049880 x1875449863912448/t0(0) o41->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572550.510651:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863912576:192.168.202.132@tcp:38:mount.lustre.0 00000100:00080000:0.0:1788572550.511572:0:111058:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-MDT0001_UUID 00800000:01000000:0.0:1788572550.511748:0:111058:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc51d2a800(969e09c0-9457-45fe-bdd1-1282a4dee8fd) successfully (5) 00800000:00000004:0.0:1788572550.511754:0:111058:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc51d2a800: age 3306, max_age 0 00000080:00000004:0.0:1788572550.511813:0:111058:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc51d2a800: age 3306, max_age 4305 00800000:00000004:0.0:1788572550.511819:0:111058:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc51d2a800: age 3306, max_age 4305 00000100:00080000:0.0:1788572550.511843:0:111058:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704aa00 x1875449863912832/t0(0) o41->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572550.511855:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a0c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572550.511902:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:0.0:1788572550.511920:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704a680 x1875449863912704/t0(0) o41->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572550.516410:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc51d2a800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572550.516429:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc51d2a800: connect to target with instance 1 00000100:00080000:3.0:1788572550.516439:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc51d2a800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572550.516451:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:3.0:1788572550.516455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46304800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:1.0:1788572550.517545:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a5052342): req@ffff92fc6704aa00 x1875449863912832/t0(0) o41->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572550.517571:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002939d3c5): req@ffff92fc67049880 x1875449863912448/t0(0) o41->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572550.517589:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863912320:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572550.517712:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc51d2a800: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572550.517748:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc51d2a800: connect to target with instance 1 00000100:00080000:1.0:1788572550.517757:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc51d2a800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:1.0:1788572550.517768:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:1.0:1788572550.517770:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:1.0:1788572550.518305:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a0c0 going to sleep for 11 seconds 00000100:00080000:3.0:1788572550.519212:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc6704a680 x1875449863912704/t0(0) o41->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572550.519237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863912576:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788572550.525099:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:0:1875449863912704:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572550.525148:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:0:1875449863912448:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572550.525197:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863912832:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:0.0:1788572550.525213:0:111058:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc51d2a800: update ffff92fc70030458 cache blocks 317011/353529 objects 1023700/1024000 00000080:00000004:0.0:1788572550.525220:0:111058:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc51d2a800: update ffff92fc700337d8 cache blocks 317011/353529 objects 1023700/1024000 00000100:00100000:0.0:1788572550.525263:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863912960:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:0.0:1788572550.525336:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a780 going to sleep for 11 seconds 00000100:00100000:0.0:1788572550.532325:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863912960:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:0.0:1788572550.532356:0:111058:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:0.0:1788572550.532369:0:111058:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:0.0:1788572550.532373:0:111058:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:0.0:1788572550.532386:0:111058:0:(genops.c:1385:class_connect()) connect: client 969e09c0-9457-45fe-bdd1-1282a4dee8fd, cookie 0x58a604dcef3529c6 00000020:00000080:0.0:1788572550.532405:0:111058:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef3529cd 00000100:00080000:0.0:1788572550.532414:0:111058:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc51d2a800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572550.532421:0:111058:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc51d2a800: import ffff92fc492a3800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572550.532436:0:111058:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572550.532463:0:111058:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449863913088/t0(0) o8->lustre-OST0000-osc-ffff92fc51d2a800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572550.532487:0:111058:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-OST0000_UUID 00020000:01000000:0.0:1788572550.532510:0:111058:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc51d2a800) active 00000020:00000080:0.0:1788572550.532559:0:111058:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef3529d4 00000100:00080000:0.0:1788572550.532566:0:111058:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc51d2a800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572550.532571:0:111058:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc51d2a800: import ffff92fc492a5000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572550.532577:0:111058:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572550.532589:0:111058:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449863913216/t0(0) o8->lustre-OST0001-osc-ffff92fc51d2a800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572550.532598:0:111058:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-OST0001_UUID 00020000:01000000:0.0:1788572550.532604:0:111058:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc51d2a800) active 00000100:00100000:0.0:1788572550.532641:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863913344:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572550.532708:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.533597:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863913088:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572550.533677:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863913216:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572550.543329:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863913344:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572550.543420:0:111058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863913472:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572550.543556:0:111058:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572550.544583:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc51d2a800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572550.544595:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc51d2a800: connect to target with instance 1 00000100:00080000:3.0:1788572550.544605:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc51d2a800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572550.544617:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:3.0:1788572550.544621:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc492a3800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572550.544628:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:3.0:1788572550.544634:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572550.544640:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572550.544655:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863913088:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572550.544681:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc51d2a800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572550.544686:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc51d2a800: connect to target with instance 1 00000100:00080000:3.0:1788572550.544690:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc51d2a800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572550.544694:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:3.0:1788572550.544696:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc492a5000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572550.544700:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:3.0:1788572550.544701:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572550.544704:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572550.544709:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863913216:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572550.554012:0:111058:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111058:1875449863913472:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788572550.554385:0:111058:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00200000:0.0:1788572550.554395:0:111058:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:0.0:1788572550.554469:0:111058:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:0.0:1788572550.554474:0:111058:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00010000:00010000:2.0:1788572550.854803:0:111112:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:2.0:1788572550.854856:0:111112:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc51d2a800 00010000:00010000:2.0:1788572550.854899:0:111112:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc51d2a800 00010000:00010000:2.0:1788572550.856347:0:111112:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc51d2a800 00010000:00010000:2.0:1788572550.856381:0:111112:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc51d2a800 00010000:00010000:0.0:1788572551.300904:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572551.609659:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:3.0:1788572551.609700:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572551.609803:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.609815:0:111119:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.609825:0:111119:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572551.609851:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863913600:192.168.202.132@tcp:101:find.0 00000100:00100000:3.0:1788572551.610215:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572551.614311:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863913600:192.168.202.132@tcp:101:find.0 00010000:00010000:3.0:1788572551.614339:0:111119:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614351:0:111119:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614363:0:111119:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614379:0:111119:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614393:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614402:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614410:0:111119:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572551.614423:0:111119:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733b100 x1875449863913600/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'find.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572551.614438:0:111119:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614452:0:111119:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572551.614459:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572551.614485:0:111119:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572551.614489:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:3.0:1788572551.614494:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:3.0:1788572551.614497:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e0382e9bb58 00010000:00010000:3.0:1788572551.614500:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614509:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572551.614534:0:111119:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:3.0:1788572551.614555:0:111119:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448), flags=11 00000080:00200000:3.0:1788572551.614560:0:111119:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448), intent=getattr 00000002:00010000:3.0:1788572551.614581:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59c.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572551.614619:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614627:0:111119:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572551.614637:0:111119:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572551.614669:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863913728:192.168.202.132@tcp:101:find.0 00000100:00100000:3.0:1788572551.614771:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572551.619452:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863913728:192.168.202.132@tcp:101:find.0 00010000:00010000:3.0:1788572551.619479:0:111119:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619601:0:111119:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619611:0:111119:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619623:0:111119:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619637:0:111119:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619664:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619672:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619679:0:111119:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.619692:0:111119:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733b100 x1875449863913728/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'find.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572551.619705:0:111119:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619715:0:111119:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589935649/1 (0 0) 00000080:00200000:2.0:1788572551.619756:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572551.619791:0:111119:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00200000:2.0:1788572551.619799:0:111119:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572551.619814:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef3529e2 00000080:00010000:2.0:1788572551.619825:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572551.619828:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529e2 from it ffff9e0382e9bb20 00010000:00010000:2.0:1788572551.619832:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.619841:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572551.619886:0:111119:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572551.619893:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88),name=d59c.sanity-sec 00000100:00100000:2.0:1788572551.620127:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863913856:192.168.202.132@tcp:900:find.0 00000100:00100000:2.0:1788572551.620220:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.623380:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863913856:192.168.202.132@tcp:900:find.0 00000002:00010000:2.0:1788572551.623426:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.623446:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572551.623461:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572551.623472:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.623482:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572551.623486:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef3529e2 00000080:00010000:2.0:1788572551.623494:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529e2 from it ffff9e0382e9bcb8 00010000:00010000:2.0:1788572551.623498:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572551.623580:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572551.623815:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572551.623830:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.623839:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572551.623848:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572551.623854:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.623858:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572551.623861:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572551.623865:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e0382e9bbf8 00010000:00010000:2.0:1788572551.623868:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572551.624004:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572551.624113:0:111119:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.624120:0:111119:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=267 00000080:00200000:2.0:1788572551.624126:0:111119:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572551.624135:0:111119:0:(file.c:1090:ll_file_open()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), flags 704400 00000002:00010000:2.0:1788572551.624152:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788572551.624210:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.624219:0:111119:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572551.624234:0:111119:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572551.624244:0:111119:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ed80 x1875449863914112/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572551.624253:0:111119:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.624273:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914112:192.168.202.132@tcp:101:find.0 00000100:00100000:2.0:1788572551.624450:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.630358:0:111119:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc51d2a800: committing for last_committed 4294983757 gen 1 00000100:00100000:2.0:1788572551.630379:0:111119:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617ed80 x1875449863914112/t4294983757(4294983757) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 3 fl Rpc:RPQU/604/0 rc 0/301 job:'find.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572551.630391:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914112:192.168.202.132@tcp:101:find.0 00010000:00010000:2.0:1788572551.630410:0:111119:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630426:0:111119:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630432:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630437:0:111119:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630446:0:111119:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630453:0:111119:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366600 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572551.630456:0:111119:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630462:0:111119:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630468:0:111119:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630474:0:111119:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572551.630480:0:111119:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.630486:0:111119:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529e9 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.630504:0:111119:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617ed80 x1875449863914112/t4294983757(4294983757) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'find.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572551.630521:0:111119:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ed80 x1875449863914112/t4294983757(4294983757) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'find.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.630532:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572551.630552:0:111119:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572551.630574:0:111119:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ed80 x1875449863914112/t4294983757(4294983757) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'find.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572551.630633:0:111119:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572551.630640:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88),name=d59c.sanity-sec 00000002:00010000:2.0:1788572551.630835:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.630851:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572551.630862:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572551.631064:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.631071:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572551.631075:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef3529e2 00000080:00010000:2.0:1788572551.631080:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529e2 from it ffff9e0382e9bcd8 00010000:00010000:2.0:1788572551.631084:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572551.631094:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572551.631204:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572551.631214:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.631222:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572551.631231:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572551.631237:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.631241:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572551.631243:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572551.631247:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e0382e9bb58 00010000:00010000:2.0:1788572551.631250:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572551.631257:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572551.631265:0:111119:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.631273:0:111119:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=11 00000080:00200008:2.0:1788572551.631280:0:111119:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572551.631283:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88),name=d59c.sanity-sec 00000002:00010000:2.0:1788572551.631291:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.631297:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572551.631304:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572551.631315:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.631319:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572551.631321:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef3529e2 00000080:00010000:2.0:1788572551.631324:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529e2 from it ffff9e0382e9bcb8 00010000:00010000:2.0:1788572551.631328:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572551.631334:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572551.631356:0:111119:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788572551.631398:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572551.631405:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572551.631412:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000100:00100000:2.0:1788572551.634565:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914240:192.168.202.132@tcp:37:find.0 00000100:00100000:2.0:1788572551.634686:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.648638:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914240:192.168.202.132@tcp:37:find.0 00010000:00010000:2.0:1788572551.649320:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572551.649335:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572551.649407:0:111119:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:2.0:1788572551.649504:0:111119:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.649519:0:111119:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), flags=0 00000080:00200000:2.0:1788572551.649524:0:111119:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), intent=getattr 00000002:00010000:2.0:1788572551.649563:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572551.649610:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.649619:0:111119:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.649628:0:111119:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111119 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.649648:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914368:192.168.202.132@tcp:101:find.0 00000100:00100000:2.0:1788572551.649706:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.653528:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914368:192.168.202.132@tcp:101:find.0 00010000:00010000:2.0:1788572551.653558:0:111119:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653581:0:111119:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653589:0:111119:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653605:0:111119:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653617:0:111119:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653629:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653636:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653642:0:111119:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.653671:0:111119:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863914368/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/720 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'find.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572551.653684:0:111119:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653696:0:111119:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589935649/2 (0 0) 00000080:00200000:2.0:1788572551.653702:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572551.653773:0:111119:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e6c08) 00000080:00200000:2.0:1788572551.653784:0:111119:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x2:0x0], updating i_size 4096 00000080:00010000:2.0:1788572551.653799:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x2:0x0](ffff92fc665e6c08) for lock 0x58a604dcef3529f0 00000080:00010000:2.0:1788572551.653811:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x2:0x0](ffff92fc665e6c08) 00000080:00010000:2.0:1788572551.653816:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529f0 from it ffff9e0382e9bb20 00010000:00010000:2.0:1788572551.653819:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.653827:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572551.653867:0:111119:0:(file.c:6296:ll_getattr_dentry()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x200000421:0x2:0x0](ffff92fc665e6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572551.655125:0:111119:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e6c08),name=h5QWF6z1nysTh0IQ8KjNGg 00000002:00010000:2.0:1788572551.655184:0:111119:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x2:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.655198:0:111119:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572551.655212:0:111119:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572551.655219:0:111119:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.655224:0:111119:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x2:0x0](ffff92fc665e6c08) 00000080:00010000:2.0:1788572551.655228:0:111119:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x2:0x0](ffff92fc665e6c08) for lock 0x58a604dcef3529f0 00000080:00010000:2.0:1788572551.655233:0:111119:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529f0 from it ffff9e0382e9bcb8 00010000:00010000:2.0:1788572551.655236:0:111119:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572551.655244:0:111119:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788572551.655380:0:111119:0:(file.c:479:ll_file_release()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), flags 704000 00000002:00100000:2.0:1788572551.655438:0:111119:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617ed80 x1875449863914112/t4294983757(4294983757) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'find.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572551.655451:0:111119:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572551.655456:0:111119:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863914624/t0(0) o35->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572551.655477:0:111119:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914624:192.168.202.132@tcp:35:find.0 00000100:00100000:2.0:1788572551.655562:0:111119:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fed80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.663377:0:111119:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job find:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111119:1875449863914624:192.168.202.132@tcp:35:find.0 00000002:00100000:2.0:1788572551.663420:0:111119:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617ed80 x1875449863914112/t4294983757(4294983757) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'find.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.876442:0:111122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572551.876472:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.876490:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572551.876506:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572551.876514:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.876523:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572551.876526:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572551.876533:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e03823ffbf8 00010000:00010000:2.0:1788572551.876537:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572551.876544:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572551.876565:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.876574:0:111122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=27 00000080:00200000:2.0:1788572551.876581:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.876585:0:111122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, flags=267 00000080:00200000:2.0:1788572551.876616:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e6c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572551.876625:0:111122:0:(file.c:1090:ll_file_open()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x200000421:0x2:0x0](ffff92fc665e6c08), flags 304000 00000002:00010000:2.0:1788572551.876639:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x2:0x0]) in obj [0x200000421:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:2.0:1788572551.877057:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.877065:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572551.877075:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572551.877087:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863914880/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572551.877096:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.877118:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863914880:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.877328:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.882140:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863914880:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.882175:0:111122:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882188:0:111122:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882201:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882209:0:111122:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882218:0:111122:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882226:0:111122:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572551.882231:0:111122:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882239:0:111122:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882246:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 5/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882253:0:111122:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572551.882259:0:111122:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.882265:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef3529f7 lrc: 0/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.882282:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617ea00 x1875449863914880/t4294983759(4294983759) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572551.882294:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ea00 x1875449863914880/t4294983759(4294983759) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.882309:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572551.882334:0:111122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x2:0x0], updating i_size 4096 00000002:00100000:2.0:1788572551.882351:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ea00 x1875449863914880/t4294983759(4294983759) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.882432:0:111122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572551.882448:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.882461:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572551.882473:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572551.882480:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.882486:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572551.882489:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572551.882494:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e03823ffbf8 00010000:00010000:2.0:1788572551.882497:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572551.882506:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572551.882520:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.882527:0:111122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=24 00000080:00200000:2.0:1788572551.882531:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.882535:0:111122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, flags=264 00000080:00200000:2.0:1788572551.882565:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e6c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572551.882573:0:111122:0:(file.c:1090:ll_file_open()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x200000421:0x2:0x0](ffff92fc665e6c08), flags 504000 00000080:00200008:2.0:1788572551.882592:0:111122:0:(file.c:479:ll_file_release()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x200000421:0x2:0x0](ffff92fc665e6c08), flags 304000 00000080:00200008:2.0:1788572551.882650:0:111122:0:(file.c:479:ll_file_release()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x200000421:0x2:0x0](ffff92fc665e6c08), flags 504000 00000002:00100000:2.0:1788572551.882893:0:111122:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617ea00 x1875449863914880/t4294983759(4294983759) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572567 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572551.882965:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572551.882971:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ed80 x1875449863915136/t0(0) o35->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572551.882990:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915136:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572551.883186:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.892819:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915136:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572551.892855:0:111122:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617ea00 x1875449863914880/t4294983759(4294983759) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.892953:0:111122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572551.892972:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.892991:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572551.893003:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572551.893011:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.893018:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572551.893021:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572551.893029:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e03823ffbf8 00010000:00010000:2.0:1788572551.893033:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572551.893040:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572551.893053:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.893061:0:111122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=267 00000080:00200000:2.0:1788572551.893068:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572551.893075:0:111122:0:(file.c:1090:ll_file_open()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), flags 304000 00000002:00010000:2.0:1788572551.893089:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:2.0:1788572551.893133:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.893140:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572551.893151:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572551.893157:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ed80 x1875449863915392/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572551.893177:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.893195:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915392:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.893363:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.896259:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915392:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.896284:0:111122:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896301:0:111122:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896307:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896314:0:111122:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896322:0:111122:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896327:0:111122:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367000 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572551.896331:0:111122:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896337:0:111122:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896343:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896349:0:111122:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572551.896355:0:111122:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.896361:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef3529fe lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.896378:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617ed80 x1875449863915392/t4294983761(4294983761) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572551.896396:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ed80 x1875449863915392/t4294983761(4294983761) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.896409:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572551.896429:0:111122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572551.896448:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ed80 x1875449863915392/t4294983761(4294983761) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572551.896537:0:111122:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88) cmd=800466f7 arg=7ffe33c7b230 00000080:00200000:2.0:1788572551.896552:0:111122:0:(file.c:5885:ll_migrate()) migrate [0x200000421:0x1:0x0]/h5QWF6z1nysTh0IQ8KjNGg to MDT0001 stripe count 0 00000100:00100000:2.0:1788572551.896637:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915520:192.168.202.132@tcp:700:lfs.0 00000100:00100000:2.0:1788572551.896712:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.899478:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915520:192.168.202.132@tcp:700:lfs.0 00000100:00100000:2.0:1788572551.899543:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915648:192.168.202.132@tcp:900:lfs.0 00000100:00100000:2.0:1788572551.899607:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.901628:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863915648:192.168.202.132@tcp:900:lfs.0 00010000:00010000:2.0:1788572551.901689:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572551.901845:0:111122:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572551.901854:0:111122:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572551.901886:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70205a00/0x58a604dcef3529e2 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ca2 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572551.901923:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788572551.901969:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572551.901986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863915776:192.168.202.132@tcp:103:lfs.0 00010000:00010000:2.0:1788572551.902079:0:111122:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572551.902086:0:111122:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572551.902097:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef3529f0 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ccc expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788572551.902124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863915904:192.168.202.132@tcp:103:lfs.0 00000020:00100000:2.0:1788572551.902211:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572551.902220:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863916160/t0(0) o36->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:13/10 lens 704/568 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572551.902238:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916160:192.168.202.132@tcp:36:lfs.0 00000100:00100000:2.0:1788572551.902265:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.905619:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863915904:192.168.202.132@tcp:103:lfs.0 00000100:00100000:2.0:1788572551.905637:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863915776:192.168.202.132@tcp:103:lfs.0 00000100:00100000:2.0:1788572551.945026:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916160:192.168.202.132@tcp:36:lfs.0 00000080:00200000:2.0:1788572551.945072:0:111122:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e6c08) 00000080:00200000:2.0:1788572551.945173:0:111122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572551.945199:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.945215:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572551.945233:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572551.945241:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572551.945247:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572551.945251:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572551.945258:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e03823ffbf8 00010000:00010000:2.0:1788572551.945262:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572551.945270:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572551.945281:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.945302:0:111122:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448), flags=27 00000080:00200000:2.0:1788572551.945308:0:111122:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448), intent=0 00000002:00010000:2.0:1788572551.945320:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59c.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572551.945358:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.945364:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.945373:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.945395:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916288:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.945469:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.951149:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916288:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.951186:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951208:0:111122:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951216:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951228:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951239:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951251:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951259:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951266:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.951292:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863916288/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572551.951304:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951314:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589935649/1 (0 0) 00000080:00200000:2.0:1788572551.951320:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572551.951340:0:111122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1788572551.951342:0:111122:0:(namei.c:147:ll_iget()) got inode: [0x200000421:0x1:0x0](ffff92fc665e5b88): rc = 0 00000080:00010000:2.0:1788572551.951352:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef352a05 00000080:00010000:2.0:1788572551.951365:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572551.951368:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a05 from it ffff9e03823ffac8 00010000:00010000:2.0:1788572551.951372:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951382:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572551.951417:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572551.951434:0:111122:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), file ffff92fc49155980, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572551.951440:0:111122:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), intent=open 00000002:00010000:2.0:1788572551.951491:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572551.951531:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.951538:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572551.951549:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572551.951555:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863916544/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572551.951564:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.951581:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916544:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.951771:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.963302:0:111122:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc51d2a800: committing for last_committed 4294983761 gen 1 00000100:00100000:2.0:1788572551.963322:0:111122:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617ed80 x1875449863915392/t4294983761(4294983761) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572551.963337:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916544:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.963362:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963382:0:111122:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963392:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963405:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963416:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963431:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963441:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963449:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.963461:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863916544/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572551.963473:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963483:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 9663677450/1 (0 0) 00000002:00010000:2.0:1788572551.963498:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x24000040a:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572551.963551:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.963559:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572551.963569:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572551.963575:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c700 x1875449863916800/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/131800 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572551.963582:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.963601:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916800:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.964370:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.977809:0:111122:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc51d2a800: committing for last_committed 4294976754 gen 1 00000100:00100000:2.0:1788572551.977838:0:111122:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc4617c700 x1875449863916800/t4294976754(4294976754) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/784 e 0 to 0 dl 1788572567 ref 3 fl Rpc:RPQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572551.977850:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916800:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.977866:0:111122:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977877:0:111122:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977885:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977891:0:111122:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 3/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977899:0:111122:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 4/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977917:0:111122:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572551.977922:0:111122:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 4/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977927:0:111122:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 4/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977938:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 5/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977945:0:111122:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572551.977951:0:111122:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 3/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.977957:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a13 lrc: 0/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.977982:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617c700 x1875449863916800/t4294976754(4294976754) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/784 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572551.977994:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c700 x1875449863916800/t4294976754(4294976754) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/784 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572551.978051:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572551.978097:0:111122:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08) 00000002:00010000:2.0:1788572551.978123:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.978178:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.978186:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.978193:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.978214:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916928:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.978287:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.994707:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863916928:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.994774:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 4/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994786:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 4/1,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994796:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994823:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994836:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994844:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994850:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.994862:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863916928/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/728 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572551.994875:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994888:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677440/5 (0 0) 00010000:00010000:2.0:1788572551.994896:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994905:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572551.994944:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x2:0x0]) in obj [0x200000421:0x2:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572551.994991:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.994998:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.995007:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572551.995027:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917056:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572551.995103:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572551.998887:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917056:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572551.999068:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999080:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999091:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999101:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999113:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999120:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999128:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572551.999140:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863917056/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/728 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572551.999153:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999176:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589935649/2 (0 0) 00010000:00010000:2.0:1788572551.999185:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999192:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572551.999227:0:111122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 8192 00000080:00010000:2.0:1788572551.999250:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352a0c 00000080:00010000:2.0:1788572551.999286:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) 00000080:00200008:2.0:1788572551.999301:0:111122:0:(file.c:1090:ll_file_open()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), flags 304000 00000002:00100000:2.0:1788572551.999322:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c700 x1875449863916800/t4294976754(4294976754) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 608/784 e 0 to 0 dl 1788572567 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572551.999366:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a0c from it ffff92fc672d12a0 00010000:00010000:2.0:1788572551.999371:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572551.999378:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572551.999391:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572551.999418:0:111122:0:(file.c:479:ll_file_release()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e5b88), flags 304000 00000002:00100000:2.0:1788572551.999462:0:111122:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617ed80 x1875449863915392/t4294983761(4294983761) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572551.999472:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572551.999477:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863917312/t0(0) o35->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572551.999497:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917312:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572551.999625:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.005319:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917312:192.168.202.132@tcp:35:lfs.0 00000002:00100000:2.0:1788572552.005352:0:111122:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617ed80 x1875449863915392/t4294983761(4294983761) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572552.005441:0:111122:0:(file.c:6296:ll_getattr_dentry()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572552.005448:0:111122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08),name=h5QWF6z1nysTh0IQ8KjNGg 00000002:00010000:2.0:1788572552.005507:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572552.005564:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.005578:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.005587:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.005606:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917440:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572552.006315:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.009409:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917440:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572552.009436:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009465:0:111122:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009472:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009483:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009492:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009503:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009516:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009523:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.009534:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ea00 x1875449863917440/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/760 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.009548:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009558:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677450/1 (0 0) 00000080:00200000:2.0:1788572552.009563:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572552.009595:0:111122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572552.009601:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352a28 00000080:00010000:2.0:1788572552.009605:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) 00000080:00010000:2.0:1788572552.009608:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a28 from it ffff9e03823ffcd8 00010000:00010000:2.0:1788572552.009611:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.009619:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572552.009646:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572552.009655:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.009669:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.009676:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.009683:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000002:00010000:2.0:1788572552.009689:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x2:0x0]) in obj [0x200000421:0x2:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572552.009695:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.009704:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.009710:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.009716:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572552.009855:0:111122:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08) pos/size0/8192 32bit_api 0 00010000:00010000:2.0:1788572552.009960:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572552.009989:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572552.009997:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572552.010240:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000100:00100000:2.0:1788572552.011585:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917568:192.168.202.132@tcp:37:lfs.0 00000100:00100000:2.0:1788572552.011817:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.022541:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917568:192.168.202.132@tcp:37:lfs.0 00010000:00010000:2.0:1788572552.023585:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572552.023604:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000002:00010000:2.0:1788572552.023630:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x2:0x0]) in obj [0x200000421:0x2:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572552.023646:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572552.023890:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788572552.024991:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917696:192.168.202.132@tcp:37:lfs.0 00000100:00100000:2.0:1788572552.025212:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.034065:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917696:192.168.202.132@tcp:37:lfs.0 00010000:00010000:2.0:1788572552.034783:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572552.034800:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200080:2.0:1788572552.034909:0:111122:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08) cmd=800466f7 arg=7ffe33c7b230 00000080:00200000:2.0:1788572552.034924:0:111122:0:(file.c:5885:ll_migrate()) migrate [0x24000040a:0x1:0x0]/OOtV20RBTn4H_S6zIbG0AQ to MDT0001 stripe count 0 00000100:00100000:2.0:1788572552.035008:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917824:192.168.202.132@tcp:34:lfs.0 00000100:00100000:2.0:1788572552.035082:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.039501:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863917824:192.168.202.132@tcp:34:lfs.0 00000080:00200000:2.0:1788572552.039553:0:111122:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x3:0x0](ffff92fc665e0908) 00000002:00010000:2.0:1788572552.040029:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x3:0x0]) in obj [0x200000421:0x3:0x0], intent: open flags 01004040000002 00010000:00010000:2.0:1788572552.040088:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 3/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.040098:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x81000000001000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 3/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572552.040109:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572552.040118:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ea00 x1875449863918080/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572552.040128:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 3/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x81000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.040149:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863918080:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572552.040375:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.046967:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863918080:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572552.046993:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 4/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047136:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 4/1,0 mode: --/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047150:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 4/1,0 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047172:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 4/1,0 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047186:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 5/1,1 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047194:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 5/1,1 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047201:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 4/0,1 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.047212:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x600002b, status=0 req@ffff92fc4617ea00 x1875449863918080/t4294983764(4294983764) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572568 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572552.047224:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ea00 x1875449863918080/t4294983764(4294983764) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572568 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.047239:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 3/0,1 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x81000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047265:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 2 res 8589935649/3 (0 0) 00000080:00200000:2.0:1788572552.047271:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000421:0x3:0x0] suppgids 0 0: rc 0 00000002:00100000:2.0:1788572552.047289:0:111122:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ea00 x1875449863918080/t4294983764(4294983764) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572568 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572552.047298:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x3:0x0](ffff92fc665e0908) for lock 0x58a604dcef352a2f 00000080:00010000:2.0:1788572552.047303:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a2f from it ffff9e03823ffb38 00010000:00010000:2.0:1788572552.047307:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 3/0,1 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047314:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 2/0,0 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 2 type: IBT gid 0 flags: 0x91000000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788572552.047357:0:111122:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00010000:00010000:2.0:1788572552.047368:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 31 res 8589935649/3 (0 0) 00000080:00010000:2.0:1788572552.047376:0:111122:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000421:0x3:0x0](ffff92fc665e0908) 00000002:00010000:2.0:1788572552.047384:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x3:0x0]) in obj [0x200000421:0x3:0x0], intent: layout flags 0 00010000:00010000:2.0:1788572552.047421:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047428:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.047436:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.047454:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863918208:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572552.047526:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fea80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.050353:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863918208:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572552.050386:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 4/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.050400:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 4/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.050414:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff92fc4617ed80 x1875449863918208/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 376/400 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.050424:0:111122:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572552.050436:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.050448:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 16 res 8589935649/3 (0 0) 00000080:00200000:2.0:1788572552.050454:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000421:0x3:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572552.050471:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x3:0x0](ffff92fc665e0908) for lock 0x58a604dcef352a36 00000080:00010000:2.0:1788572552.050479:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a36 from it ffff9e03823ff9e8 00010000:00010000:2.0:1788572552.050482:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.050493:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.050517:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.050528:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572552.050536:0:111122:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000421:0x3:0x0](00000000f7500ab6) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.050596:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.050605:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572552.050631:0:111122:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572552.050639:0:111122:0:(vvp_io.c:304:vvp_io_fini()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1788572552.050668:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572552.050678:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788572552.050685:0:111122:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000421:0x3:0x0](00000000f7500ab6) being reconfigured ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572552.050715:0:111122:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788572552.050764:0:111122:0:(vvp_io.c:304:vvp_io_fini()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572552.050801:0:111122:0:(file.c:6843:ll_layout_conf()) [0x200000421:0x3:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788572552.050818:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572552.050828:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00020000:00200000:2.0:1788572552.050844:0:111122:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788572552.050925:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572552.050942:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863918336:192.168.202.132@tcp:1:lfs.0 00000100:00100000:2.0:1788572552.059709:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863918336:192.168.202.132@tcp:1:lfs.0 00010000:00010000:0.0:1788572552.060312:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572552.060370:0:111122:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572552.060378:0:111122:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572552.060413:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a28 lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792dac expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572552.060438:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1788572552.060507:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572552.060522:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2393:1875449863918464:192.168.202.132@tcp:103:lfs.0 00010000:00010000:0.0:1788572552.060623:0:111122:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572552.060630:0:111122:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572552.060639:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352a21 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x2:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792d9e expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1788572552.060761:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572552.060771:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863918592:192.168.202.132@tcp:103:lfs.0 00010000:00010000:0.0:1788572552.062352:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572552.062473:0:111122:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 1/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572552.062481:0:111122:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572552.062497:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352a1a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792d90 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00100000:0.0:1788572552.062527:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572552.062544:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2393:1875449863918720:192.168.202.132@tcp:103:lfs.0 00000100:00100000:0.0:1788572552.063664:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2393:1875449863918464:192.168.202.132@tcp:103:lfs.0 00010000:00010000:0.0:1788572552.063718:0:111122:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 2/0,0 mode: PW/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x81008000000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572552.063756:0:111122:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367e00/0x58a604dcef352a2f lrc: 0/0,0 mode: --/PW res: [0x200000421:0x3:0x0].0x0 bits 0x4/0x0 rrc: 3 type: IBT gid 0 flags: 0x85809400000000 nid: local remote: 0xc9a239a221792dcf expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572552.063770:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572552.063778:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704a680 x1875449863918976/t0(0) o36->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:13/10 lens 704/736 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572552.063796:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863918976:192.168.202.132@tcp:36:lfs.0 00000100:00100000:1.0:1788572552.063922:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863918592:192.168.202.132@tcp:103:lfs.0 00000100:00100000:1.0:1788572552.066776:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ae40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572552.070593:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2393:1875449863918720:192.168.202.132@tcp:103:lfs.0 00000100:00100000:0.0:1788572552.079428:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572552.079873:0:111059:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881891456 00000100:00100000:1.0:1788572552.079884:0:111059:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881891456 00000100:00100000:1.0:1788572552.079946:0:111059:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:1.0:1788572552.079956:0:111059:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67048e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:6542:x1875449881891456:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572552.079968:0:111059:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x58a604dcef352a2f - lock disappeared 00010000:00010000:1.0:1788572552.080037:0:111059:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.202.132@tcp lock=0x58a604dcef352a2f: rc = 0 req@ffff92fc67048e00 x1875449881891456/t0(0) o104->969e09c0-9457-45fe-bdd1-1282a4dee8fd@192.168.202.132@tcp:213/0 lens 328/224 e 0 to 0 dl 1788572563 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572552.080049:0:111059:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67048e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:6542:x1875449881891456:12345-192.168.202.132@tcp:104: Request processed in 96us (634us total) trans 0 rc -22/-22 00000100:00100000:1.0:1788572552.080055:0:111059:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 0 00000100:00100000:1.0:1788572552.089180:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863918976:192.168.202.132@tcp:36:lfs.0 00000002:00100000:1.0:1788572552.089220:0:111122:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617ea00 x1875449863918080/t4294983764(4294983764) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572552.089291:0:111122:0:(vvp_io.c:1826:vvp_io_init()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572552.089303:0:111122:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572552.089341:0:111122:0:(vvp_io.c:304:vvp_io_fini()) [0x200000421:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572552.089348:0:111122:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x3:0x0](ffff92fc665e0908) 00000080:00200000:1.0:1788572552.089479:0:111122:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08) pos/size18446744073709551614/8192 32bit_api 0 00000080:00200008:1.0:1788572552.089523:0:111122:0:(file.c:479:ll_file_release()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), flags 304000 00000002:00100000:1.0:1788572552.089574:0:111122:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617c700 x1875449863916800/t4294976754(4294976754) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 608/784 e 0 to 0 dl 1788572567 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572552.089585:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572552.089591:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6728b480 x1875449863919232/t0(0) o35->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572552.089613:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919232:192.168.202.132@tcp:35:lfs.0 00000100:00100000:1.0:1788572552.089704:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfe40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572552.099072:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919232:192.168.202.132@tcp:35:lfs.0 00000002:00100000:1.0:1788572552.099111:0:111122:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617c700 x1875449863916800/t4294976754(4294976754) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 608/784 e 0 to 0 dl 1788572567 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572552.099198:0:111122:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:1.0:1788572552.099218:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572552.099237:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572552.099251:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572552.099259:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572552.099265:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:1.0:1788572552.099268:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:1.0:1788572552.099275:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e03823ffbe8 00010000:00010000:1.0:1788572552.099279:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788572552.099287:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572552.099302:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:1.0:1788572552.099309:0:111122:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=25 00000080:00200000:1.0:1788572552.099314:0:111122:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572552.099326:0:111122:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), flags=9 00000080:00200000:1.0:1788572552.099330:0:111122:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), intent=getattr 00000002:00010000:1.0:1788572552.099373:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572552.099412:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.099420:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.099430:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572552.099449:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919360:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572552.099522:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfe40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572552.102416:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919360:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572552.102442:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102458:0:111122:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102465:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102476:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102491:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102506:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102514:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102522:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572552.102535:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b480 x1875449863919360/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572552.102547:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102559:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572552.102571:0:111122:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572552.102580:0:111122:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102586:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102593:0:111122:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102618:0:111122:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:1.0:1788572552.102637:0:111122:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040a:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572552.102696:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102705:0:111122:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.102713:0:111122:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111122 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572552.102767:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919488:192.168.202.132@tcp:101:lfs.0 00000100:00100000:1.0:1788572552.102827:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccf000 going to sleep for 11 seconds 00010000:00010000:2.0:1788572552.102893:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572552.102913:0:111069:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.102921:0:111069:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f4400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572552.102926:0:111069:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.102934:0:111069:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.102942:0:111069:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 3/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.102960:0:111069:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.102993:0:111069:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 2/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.103001:0:111069:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572552.103029:0:111069:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.103038:0:111069:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f4400/0x58a604dcef352a3d lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792e38 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.103051:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572552.103053:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572552.103055:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572552.103168:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572552.103186:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863919616:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572552.106232:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863919616:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572552.106331:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919488:192.168.202.132@tcp:101:lfs.0 00010000:00010000:1.0:1788572552.106351:0:111122:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106363:0:111122:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106376:0:111122:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106389:0:111122:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106402:0:111122:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106411:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106419:0:111122:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572552.106430:0:111122:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67289500 x1875449863919488/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/760 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572552.106443:0:111122:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106455:0:111122:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677450/1 (0 0) 00000080:00200000:1.0:1788572552.106467:0:111122:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572552.106500:0:111122:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572552.106503:0:111122:0:(namei.c:147:ll_iget()) got inode: [0x24000040a:0x1:0x0](ffff92fc665e6c08): rc = 0 00000080:00010000:1.0:1788572552.106515:0:111122:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) for remote lock 0x58a604dcef352a44 00000080:00010000:1.0:1788572552.106519:0:111122:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352a0c 00000080:00010000:1.0:1788572552.106530:0:111122:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) 00000080:00010000:1.0:1788572552.106534:0:111122:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a0c from it ffff9e03823ffbb0 00010000:00010000:1.0:1788572552.106538:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572552.106547:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788572552.106561:0:111122:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x58a604dcef352a44 from it ffff9e03823ffbb0 00010000:00010000:1.0:1788572552.106564:0:111122:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572552.106572:0:111122:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572552.106606:0:111122:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08), xattr lmv 00000020:00100000:1.0:1788572552.106643:0:111122:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572552.106649:0:111122:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67289500 x1875449863919872/t0(0) o36->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 552/440 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572552.106675:0:111122:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919872:192.168.202.132@tcp:36:lfs.0 00000100:00100000:1.0:1788572552.106750:0:111122:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccf000 going to sleep for 11 seconds 00000100:00100000:0.0:1788572552.110106:0:2385:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572552.110201:0:111059:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881892992 00000100:00100000:1.0:1788572552.110212:0:111059:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881892992 00000100:00100000:1.0:1788572552.110268:0:111059:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 1 00000100:00100000:1.0:1788572552.110279:0:111059:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67049500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:8454:x1875449881892992:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572552.110300:0:111059:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572552.110387:0:111059:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572552.110394:0:111059:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67049500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:8454:x1875449881892992:12345-192.168.202.132@tcp:104: Request processed in 117us (299us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572552.110401:0:111059:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 1 00010000:00010000:3.0:1788572552.110466:0:111070:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572552.110473:0:111070:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110481:0:111070:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367000 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:3.0:1788572552.110488:0:111070:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110497:0:111070:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110506:0:111070:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 3/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110562:0:111070:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 2/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110569:0:111070:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572552.110595:0:111070:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110603:0:111070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367000/0x58a604dcef352a0c lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792d74 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.110627:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.110629:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.110631:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572552.110638:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572552.110665:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863920000:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572552.117322:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2396:1875449863920000:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572552.120951:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:1.0:1788572552.121181:0:111059:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881893248 00000100:00100000:1.0:1788572552.121192:0:111059:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881893248 00000100:00100000:1.0:1788572552.121248:0:111059:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 2 00000100:00100000:1.0:1788572552.121258:0:111059:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67049c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:44285:x1875449881893248:12345-192.168.202.132@tcp:104: 00010000:00010000:1.0:1788572552.121277:0:111059:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572552.121350:0:111059:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788572552.121357:0:111059:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67049c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:44285:x1875449881893248:12345-192.168.202.132@tcp:104: Request processed in 101us (416us total) trans 0 rc 0/0 00000100:00100000:1.0:1788572552.121365:0:111059:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 2 00010000:00010000:2.0:1788572552.121528:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572552.121536:0:111069:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121544:0:111069:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f5400 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572552.121549:0:111069:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121557:0:111069:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121566:0:111069:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 3/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121620:0:111069:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 2/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121627:0:111069:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572552.121657:0:111069:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121665:0:111069:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc702f5400/0x58a604dcef352a44 lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792e46 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572552.121684:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572552.121686:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572552.121687:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572552.121696:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572552.121710:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863920128:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572552.125068:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863920128:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572552.133999:0:111122:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111122:1875449863919872:192.168.202.132@tcp:36:lfs.0 00000080:00200000:2.0:1788572552.145121:0:111123:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572552.145151:0:111123:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572552.145183:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572552.145200:0:111123:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572552.145208:0:111123:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572552.145217:0:111123:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572552.145220:0:111123:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572552.145227:0:111123:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e0382eabbf8 00010000:00010000:2.0:1788572552.145231:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572552.145245:0:111123:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572552.145261:0:111123:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572552.145268:0:111123:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=27 00000080:00200000:2.0:1788572552.145274:0:111123:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572552.145284:0:111123:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), file ffff92fc608bfc80, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788572552.145291:0:111123:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, dir=[0x200000421:0x1:0x0](ffff92fc665e5b88), intent=open 00000002:00010000:2.0:1788572552.145343:0:111123:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572552.145396:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.145404:0:111123:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572552.145416:0:111123:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572552.145428:0:111123:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863920384/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 776/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572552.145438:0:111123:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.145458:0:111123:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920384:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572552.145842:0:111123:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fed80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.153565:0:111123:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc51d2a800: committing for last_committed 4294983765 gen 1 00000100:00100000:2.0:1788572552.153595:0:111123:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920384:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572552.153618:0:111123:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153639:0:111123:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153646:0:111123:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153670:0:111123:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153682:0:111123:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153696:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153704:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153711:0:111123:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.153752:0:111123:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=0 req@ffff92fc4617c380 x1875449863920384/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 776/608 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.153765:0:111123:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153775:0:111123:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 9663677450/1 (0 0) 00000002:00010000:2.0:1788572552.153792:0:111123:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x24000040a:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788572552.153840:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.153847:0:111123:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572552.153857:0:111123:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572552.153864:0:111123:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863920640/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/131864 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572552.153872:0:111123:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.153892:0:111123:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920640:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572552.154067:0:111123:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fed80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.157833:0:111123:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920640:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572552.157855:0:111123:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157867:0:111123:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157875:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157884:0:111123:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 3/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157893:0:111123:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 4/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157966:0:111123:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366800 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572552.157972:0:111123:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 4/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157978:0:111123:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 4/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157986:0:111123:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 5/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.157993:0:111123:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572552.157999:0:111123:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 3/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.158204:0:111123:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366800/0x58a604dcef352a52 lrc: 0/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.158236:0:111123:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617d180 x1875449863920640/t4294976758(4294976758) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/768 e 0 to 0 dl 1788572568 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572552.158249:0:111123:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863920640/t4294976758(4294976758) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/768 e 0 to 0 dl 1788572568 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572552.158313:0:111123:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788572552.158367:0:111123:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572552.158415:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.158422:0:111123:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.158433:0:111123:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111123 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.158453:0:111123:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920768:192.168.202.132@tcp:101:lfs.0 00000100:00100000:2.0:1788572552.158573:0:111123:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fed80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.161794:0:111123:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920768:192.168.202.132@tcp:101:lfs.0 00010000:00010000:2.0:1788572552.161819:0:111123:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.161831:0:111123:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 4/1,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.161842:0:111123:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.161852:0:111123:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.161863:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.161871:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.161878:0:111123:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.161894:0:111123:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617c380 x1875449863920768/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/728 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.162180:0:111123:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.162194:0:111123:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677440/5 (0 0) 00010000:00010000:2.0:1788572552.162202:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.162211:0:111123:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572552.162247:0:111123:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1788572552.162252:0:111123:0:(namei.c:147:ll_iget()) got inode: [0x24000040a:0x1:0x0](ffff92fc665e6c08): rc = 0 00000080:00010000:2.0:1788572552.162267:0:111123:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352a4b 00000080:00010000:2.0:1788572552.162304:0:111123:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665e6c08) 00000080:00200008:2.0:1788572552.162319:0:111123:0:(file.c:1090:ll_file_open()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), flags 304000 00000002:00100000:2.0:1788572552.162339:0:111123:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863920640/t4294976758(4294976758) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/768 e 0 to 0 dl 1788572568 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572552.162381:0:111123:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a4b from it ffff92fc672d1de0 00010000:00010000:2.0:1788572552.162386:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.162394:0:111123:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572552.162411:0:111123:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08), inode mode 41ed mask 44 00000080:00200000:2.0:1788572552.162449:0:111123:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572552.162462:0:111123:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572552.162472:0:111123:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572552.162481:0:111123:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572552.162488:0:111123:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572552.162492:0:111123:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572552.162495:0:111123:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef3529db 00000080:00010000:2.0:1788572552.162498:0:111123:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef3529db from it ffff9e0382eabbf8 00010000:00010000:2.0:1788572552.162501:0:111123:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572552.162507:0:111123:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572552.162516:0:111123:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572552.162523:0:111123:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=24 00000080:00200000:2.0:1788572552.162527:0:111123:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572552.162530:0:111123:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=h5QWF6z1nysTh0IQ8KjNGg, flags=264 00000080:00200000:2.0:1788572552.162545:0:111123:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572552.162549:0:111123:0:(file.c:1090:ll_file_open()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), flags 504000 00000080:00200008:2.0:1788572552.162565:0:111123:0:(file.c:479:ll_file_release()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), flags 304000 00000080:00200080:2.0:1788572552.162598:0:111123:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08) cmd=c03066f1 arg=17f5630 00000100:00100000:2.0:1788572552.162642:0:111123:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920896:192.168.202.132@tcp:33:lfs.0 00000100:00100000:2.0:1788572552.163183:0:111123:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fed80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.165814:0:111123:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863920896:192.168.202.132@tcp:33:lfs.0 00000080:00200008:2.0:1788572552.166179:0:111123:0:(file.c:479:ll_file_release()) START file h5QWF6z1nysTh0IQ8KjNGg:[0x24000040a:0x1:0x0](ffff92fc665e6c08), flags 504000 00000002:00100000:2.0:1788572552.166232:0:111123:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617d180 x1875449863920640/t4294976758(4294976758) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/768 e 0 to 0 dl 1788572568 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572552.166244:0:111123:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572552.166250:0:111123:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863921152/t0(0) o35->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572552.166270:0:111123:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863921152:192.168.202.132@tcp:35:lfs.0 00000100:00100000:2.0:1788572552.166340:0:111123:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fed80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572552.168521:0:111123:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job lfs:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111123:1875449863921152:192.168.202.132@tcp:35:lfs.0 00000002:00100000:3.0:1788572552.168553:0:111123:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617d180 x1875449863920640/t4294976758(4294976758) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/768 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788572552.243063:0:111128:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:3.0:1788572552.243104:0:111128:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc51d2a800 00010000:00010000:3.0:1788572552.243132:0:111128:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/4 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788572552.243181:0:111070:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 4 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572552.243190:0:111070:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572552.243229:0:111070:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572552.243255:0:111070:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572552.243263:0:111070:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572552.243310:0:111070:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572552.243318:0:111070:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572552.243324:0:111070:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 1/0,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572552.243330:0:111070:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572552.243336:0:111070:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00000100:00100000:0.0:1788572552.243359:0:111070:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111070:1875449863921280:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572552.243436:0:111070:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9aa80 going to sleep for 11 seconds 00010000:00010000:1.0:1788572552.243687:0:111126:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc51d2a800 00010000:00010000:1.0:1788572552.243710:0:111126:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572552.243767:0:111127:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc51d2a800 00010000:00010000:1.0:1788572552.243795:0:111127:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc51d2a800 00010000:00010000:3.0:1788572552.244002:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00000100:00100000:3.0:1788572552.246989:0:111070:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111070:1875449863921280:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572552.247034:0:111070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204000/0x58a604dcef3529db lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792c86 expref: -99 pid: 111119 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.247057:0:111070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a4b lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792e9a expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572552.247069:0:111070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367a00/0x58a604dcef352a36 lrc: 0/0,0 mode: --/CR res: [0x200000421:0x3:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792ddd expref: -99 pid: 111122 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572552.247079:0:111070:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366600/0x58a604dcef352a05 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792d58 expref: -99 pid: 111122 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572552.247108:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.247110:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.247112:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572552.248156:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572552.248170:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572552.248172:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.248199:0:111069:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572552.248268:0:111069:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572552.248276:0:111069:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572552.248298:0:111069:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111069:1875449863921408:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572552.248412:0:111069:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572552.254642:0:111069:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111069:1875449863921408:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572552.254777:0:111069:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367200/0x58a604dcef352a59 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792eb6 expref: -99 pid: 111123 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572552.254865:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.254868:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.254870:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572552.325028:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572552.368058:0:110779:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665e6c08) 00000080:00200000:2.0:1788572552.368102:0:110779:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e5b88) 00000080:00200000:2.0:1788572552.368114:0:110779:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x2:0x0](ffff92fc665e5348) 00000080:00200000:2.0:1788572552.368632:0:110779:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000400:0x5:0x0](ffff92fc665e00c8) 00000080:00200000:0.0:1788572552.736650:0:110779:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:0.0:1788572552.736680:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572552.736774:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572552.736786:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572552.736796:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572552.736827:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921536:192.168.202.132@tcp:101:bash.0 00000100:00100000:0.0:1788572552.736918:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.743604:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921536:192.168.202.132@tcp:101:bash.0 00010000:00010000:2.0:1788572552.743632:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743657:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743669:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743687:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743699:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743708:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743715:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.743849:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704ad80 x1875449863921536/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.743864:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.743878:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572552.743956:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572552.743985:0:110779:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572552.743991:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:2.0:1788572552.743997:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572552.744000:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e03820a7b58 00010000:00010000:2.0:1788572552.744004:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.744018:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572552.744044:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572552.744062:0:110779:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448), flags=25 00000080:00200000:2.0:1788572552.744066:0:110779:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448), intent=0 00000002:00010000:2.0:1788572552.744161:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d59c.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572552.744210:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.744217:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.744227:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572552.744248:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921664:192.168.202.132@tcp:101:bash.0 00000100:00100000:2.0:1788572552.744509:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fe0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572552.753321:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921664:192.168.202.132@tcp:101:bash.0 00010000:00010000:2.0:1788572552.753356:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753381:0:110779:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753389:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753402:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753412:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753424:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753432:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753440:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572552.753453:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617f800 x1875449863921664/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572552.753465:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753475:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589935649/1 (0 0) 00000080:00200000:2.0:1788572552.753481:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572552.753517:0:110779:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8) 00000080:00200000:2.0:1788572552.753524:0:110779:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572552.753540:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) for lock 0x58a604dcef352a67 00000080:00010000:2.0:1788572552.753553:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) 00000080:00010000:2.0:1788572552.753558:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a67 from it ffff9e03820a7a28 00010000:00010000:2.0:1788572552.753562:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572552.753571:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572552.753605:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572552.753624:0:110779:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e00c8), flags=25 00000080:00200000:2.0:1788572552.753628:0:110779:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e00c8), intent=0 00000002:00010000:2.0:1788572553.060467:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ‡”¬õŸ+‡Bð¨Í,[0x0:0x0:0x0]) in obj [0x200000421:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572553.060534:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.060546:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.060556:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572553.060583:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921792:192.168.202.132@tcp:101:bash.0 00000100:00100000:2.0:1788572553.061196:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fe0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572553.065834:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921792:192.168.202.132@tcp:101:bash.0 00010000:00010000:2.0:1788572553.065864:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065894:0:110779:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065907:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065919:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065930:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065942:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065950:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065958:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572553.065971:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617f800 x1875449863921792/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572569 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572553.065983:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.065992:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 9663677450/1 (0 0) 00000002:00010000:2.0:1788572553.066007:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040a:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572553.066052:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.066060:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.066069:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572553.066091:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921920:192.168.202.132@tcp:101:bash.0 00000100:00100000:2.0:1788572553.066177:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fe0c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572553.070533:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863921920:192.168.202.132@tcp:101:bash.0 00010000:00010000:0.0:1788572553.070562:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070574:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070586:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070597:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070610:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070618:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070638:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572553.070650:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc4617df80 x1875449863921920/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/744 e 0 to 0 dl 1788572569 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572553.070672:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070686:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677450/1 (0 0) 00000080:00200000:0.0:1788572553.070708:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000421:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572553.070787:0:110779:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248) 00000002:00010000:0.0:1788572553.070814:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572553.070855:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070863:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.070883:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572553.070903:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863922048:192.168.202.132@tcp:101:bash.0 00000100:00100000:0.0:1788572553.070976:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a240 going to sleep for 11 seconds 00000100:00100000:0.0:1788572553.079325:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863922048:192.168.202.132@tcp:101:bash.0 00010000:00010000:0.0:1788572553.079351:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 4/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079364:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 4/1,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079375:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079390:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079403:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079411:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079418:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572553.079427:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704b100 x1875449863922048/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/728 e 0 to 0 dl 1788572569 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572553.079451:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079464:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677440/5 (0 0) 00010000:00010000:0.0:1788572553.079471:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079479:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572553.079510:0:110779:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572553.079532:0:110779:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) for remote lock 0x58a604dcef352a75 00000080:00010000:0.0:1788572553.079537:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) for lock 0x58a604dcef352a6e 00000080:00010000:0.0:1788572553.079547:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) 00000080:00010000:0.0:1788572553.079551:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a6e from it ffff9e03820a7a28 00010000:00010000:0.0:1788572553.079555:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079564:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788572553.079573:0:110779:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x58a604dcef352a75 from it ffff9e03820a7a28 00010000:00010000:0.0:1788572553.079576:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.079583:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572553.079603:0:110779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248), inode mode 41ed mask 1 00000080:00200000:0.0:1788572553.079764:0:110779:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subf, dir=[0x24000040a:0x1:0x0](ffff92fc665eb248), flags=9 00000080:00200000:0.0:1788572553.079769:0:110779:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subf, dir=[0x24000040a:0x1:0x0](ffff92fc665eb248), intent=getattr 00000002:00010000:0.0:1788572553.115449:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 8ëUÛDAN~ý.³!±´,[0x0:0x0:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572553.115502:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.115510:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.115517:0:110779:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 110779 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572553.115539:0:110779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863922176:192.168.202.132@tcp:101:bash.0 00000100:00100000:0.0:1788572553.115621:0:110779:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a240 going to sleep for 11 seconds 00000100:00100000:0.0:1788572553.120435:0:110779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job bash:969e09c0-9457-45fe-bdd1-1282a4dee8fd:110779:1875449863922176:192.168.202.132@tcp:101:bash.0 00010000:00010000:0.0:1788572553.120462:0:110779:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 4/1,0 mode: --/CR res: [0x240000400:0x5:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120483:0:110779:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120492:0:110779:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120503:0:110779:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120514:0:110779:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120526:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120533:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120540:0:110779:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572553.120557:0:110779:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704b100 x1875449863922176/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/672 e 0 to 0 dl 1788572569 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572553.120567:0:110779:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572553.120581:0:110779:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.120591:0:110779:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677450/2 (0 0) 00000080:00200000:0.0:1788572553.120599:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x5:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572553.120640:0:110779:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040a:0x2:0x0](ffff92fc665eaa08) 00000080:00200000:0.0:1788572553.121094:0:110779:0:(file.c:6843:ll_layout_conf()) [0x24000040a:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572553.121111:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) for lock 0x58a604dcef352a83 00000080:00010000:0.0:1788572553.121123:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) 00000080:00010000:0.0:1788572553.121138:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a83 from it ffff9e03820a7b20 00010000:00010000:0.0:1788572553.121142:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.121153:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788572553.121195:0:110779:0:(file.c:6296:ll_getattr_dentry()) START file subf:[0x24000040a:0x2:0x0](ffff92fc665eaa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572553.121203:0:110779:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040a:0x2:0x0](ffff92fc665eaa08),name=subf 00000002:00010000:0.0:1788572553.121238:0:110779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x2:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572553.121248:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572553.121264:0:110779:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572553.121271:0:110779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x5:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572553.121276:0:110779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) 00000080:00010000:0.0:1788572553.121280:0:110779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) for lock 0x58a604dcef352a83 00000080:00010000:0.0:1788572553.121284:0:110779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a83 from it ffff9e03820a7cb8 00010000:00010000:0.0:1788572553.121288:0:110779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572553.121295:0:110779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572553.121315:0:110779:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040a:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788572553.121325:0:110779:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040a:0x2:0x0] 00000008:00010000:0.0:1788572553.121342:0:110779:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572553.121345:0:110779:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc702f22b0, osc lock ffff92fc45933c80, flags 0x400001002 00010000:00010000:0.0:1788572553.121391:0:110779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.121401:0:110779:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 3/1,0 mode: --/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572553.121440:0:110779:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788572553.121776:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2393:1875449863922304:192.168.202.132@tcp:101:bash.0 00010000:00010000:0.0:1788572553.125272:0:2393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.125315:0:2393:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.125331:0:2393:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.125349:0:2393:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572553.125366:0:2393:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2, setting rss=2; kms=2, end=18446744073709551615 ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572553.125379:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788572553.125387:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.125395:0:2393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.125405:0:2393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.125415:0:2393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572553.125442:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2393:1875449863922304:192.168.202.132@tcp:101:bash.0 00000080:00200000:2.0:1788572553.125602:0:110779:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040a:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:2.0:1788572553.125609:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_release():245 00000020:00010000:2.0:1788572553.125613:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572553.125617:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc702f22b0 (ffff92fc6721aab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572553.125622:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572553.125625:0:110779:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc702f22b0 (ffff92fc6721ab48/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572553.125639:0:110779:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040a:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572553.151586:0:111144:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572553.151614:0:111144:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572553.152117:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572553.152144:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572553.152153:0:111144:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572553.152162:0:111144:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572553.152166:0:111144:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:2.0:1788572553.152174:0:111144:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e038243fbf8 00010000:00010000:2.0:1788572553.152178:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572553.152187:0:111144:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572553.152201:0:111144:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572553.152209:0:111144:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=25 00000080:00200000:2.0:1788572553.152214:0:111144:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572553.152217:0:111144:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=25 00000080:00200000:2.0:1788572553.152220:0:111144:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248), inode mode 41ed mask 1 00000080:00200000:2.0:1788572553.152224:0:111144:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subf, flags=265 00000080:00200000:2.0:1788572553.152227:0:111144:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x2:0x0](ffff92fc665eaa08), inode mode 81a4 mask 44 00000080:00200008:2.0:1788572553.152233:0:111144:0:(file.c:1090:ll_file_open()) START file subf:[0x24000040a:0x2:0x0](ffff92fc665eaa08), flags 100000 00000002:00010000:2.0:1788572553.152323:0:111144:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x2:0x0]) in obj [0x24000040a:0x2:0x0], intent: open flags 0240100001 00010000:00010000:2.0:1788572553.152381:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111144 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.152389:0:111144:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111144 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572553.152401:0:111144:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572553.152414:0:111144:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617f480 x1875449863922560/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/131864 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572553.152423:0:111144:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: --/CR res: [0x24000040a:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111144 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572553.152446:0:111144:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job cat:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111144:1875449863922560:192.168.202.132@tcp:101:cat.0 00000100:00100000:2.0:1788572553.152645:0:111144:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc455fe480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572553.155674:0:111144:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job cat:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111144:1875449863922560:192.168.202.132@tcp:101:cat.0 00010000:00010000:2.0:1788572553.155703:0:111144:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.155715:0:111144:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.155949:0:111144:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.155963:0:111144:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.155988:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.155997:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.156005:0:111144:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572553.156020:0:111144:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617f480 x1875449863922560/t4294976760(4294976760) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572569 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572553.156034:0:111144:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617f480 x1875449863922560/t4294976760(4294976760) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572569 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572553.156048:0:111144:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572553.156062:0:111144:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.156078:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572553.156100:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572553.156108:0:111144:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.156114:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.156137:0:111144:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572553.156199:0:111144:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788572553.156206:0:111144:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x24000040a:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572553.156265:0:111144:0:(file.c:6843:ll_layout_conf()) [0x24000040a:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788572553.156271:0:111144:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) for lock 0x58a604dcef352a83 00000080:00010000:2.0:1788572553.156278:0:111144:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a83 from it ffff9e038243fbf0 00010000:00010000:2.0:1788572553.156282:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572553.156292:0:111144:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000002:00100000:2.0:1788572553.156320:0:111144:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617f480 x1875449863922560/t4294976760(4294976760) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572569 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572553.156387:0:111144:0:(file.c:6296:ll_getattr_dentry()) START file subf:[0x24000040a:0x2:0x0](ffff92fc665eaa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572553.156394:0:111144:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040a:0x2:0x0](ffff92fc665eaa08),name=subf 00000002:00010000:2.0:1788572553.156428:0:111144:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x2:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572553.156437:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572553.156446:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572553.156454:0:111144:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x240000400:0x5:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572553.156462:0:111144:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) 00000080:00010000:2.0:1788572553.156465:0:111144:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x2:0x0](ffff92fc665eaa08) for lock 0x58a604dcef352a83 00000080:00010000:2.0:1788572553.156469:0:111144:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a83 from it ffff9e038243fcd8 00010000:00010000:2.0:1788572553.156473:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572553.156482:0:111144:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572553.156500:0:111144:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040a:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572553.156510:0:111144:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040a:0x2:0x0] 00000008:00010000:2.0:1788572553.156528:0:111144:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572553.156532:0:111144:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550530, osc lock ffff92fc58370ab0, flags 0x400001002 00010000:00010000:2.0:1788572553.156549:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572553.156561:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788572553.156574:0:111144:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572553.156578:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572553.156587:0:111144:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572553.156597:0:111144:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572553.156606:0:111144:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040a:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:2.0:1788572553.156610:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_release():245 00000020:00010000:2.0:1788572553.156613:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572553.156616:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550530 (ffff92fc6721aab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572553.156620:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572553.156623:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550530 (ffff92fc6721ab48/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572553.156631:0:111144:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040a:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788572553.156993:0:111144:0:(file.c:2483:do_file_read_iter()) START file subf:[0x24000040a:0x2:0x0], ppos: 0, count: 4194304 00000080:00200000:2.0:1788572553.157014:0:111144:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:2.0:1788572553.157036:0:111144:0:(file.c:2090:ll_file_io_generic()) subf: read ppos: 0, bytes: 4194304 00000080:00200000:2.0:1788572553.157041:0:111144:0:(file.c:1848:ll_io_set_mirror()) subf: desiginated mirror: 0 00000020:00200000:2.0:1788572553.157047:0:111144:0:(cl_io.c:214:cl_io_rw_init()) header@00000000db94255b[0x110, 1, [0x24000040a:0x2:0x0] hash] 00000020:00200000:2.0:1788572553.157049:0:111144:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:2.0:1788572553.157052:0:111144:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040a:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572553.157064:0:111144:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:2.0:1788572553.157068:0:111144:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572553.157079:0:111144:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:2.0:1788572553.157099:0:111144:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1788572553.157102:0:111144:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550530, osc lock ffff92fc58370ab0, flags 0x0 00010000:00010000:2.0:1788572553.157112:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572553.157132:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572553.157144:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572553.157153:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572553.157161:0:111144:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572553.157170:0:111144:0:(vvp_io.c:835:vvp_io_read_start()) subf: read [0, 4194304) 00000001:00010000:2.0:1788572553.157179:0:111144:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040a:0x2:0x0] 00000008:00010000:2.0:1788572553.157184:0:111144:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572553.157186:0:111144:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc70550ab0, osc lock ffff92fc583717c0, flags 0x400001002 00010000:00010000:2.0:1788572553.157192:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572553.157201:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572553.157216:0:111144:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572553.157219:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572553.157226:0:111144:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:2.0:1788572553.157233:0:111144:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788572553.157237:0:111144:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040a:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:2.0:1788572553.157239:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_release():245 00000020:00010000:2.0:1788572553.157242:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788572553.157244:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550ab0 (ffff92fc6721aab8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788572553.157247:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_fini():72 00000020:00010000:2.0:1788572553.157249:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550ab0 (ffff92fc6721ab48/1) at cl_lock_fini():72 00000080:00200000:2.0:1788572553.157256:0:111144:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:2.0:1788572553.157291:0:111144:0:(osc_cache.c:3059:osc_queue_sync_pages()) ### no dlm lock: extent: ffff92fc58ca8840 00000008:00010000:2.0:1788572553.157324:0:110381:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc58ca8840 00010000:00010000:2.0:1788572553.157343:0:110381:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00100000:2.0:1788572553.157378:0:110381:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc4617f100 - object 0x2c0000401:538 offset 0<>4096 00000100:00100000:3.0:1788572553.157535:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863922688:192.168.202.132@tcp:3:cat.0 00010000:00010000:0.0:1788572553.158304:0:111129:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572553.158315:0:111129:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158326:0:111129:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367800 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572553.158332:0:111129:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158341:0:111129:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158350:0:111129:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 3/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158371:0:111129:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788572553.158406:0:111129:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 2/0,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158412:0:111129:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572553.158465:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863922816:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572553.158536:0:111129:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158546:0:111129:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70367800/0x58a604dcef352a91 lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f49 expref: -99 pid: 111144 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572553.158561:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572553.158563:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572553.158565:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:2.0:1788572553.166166:0:2395:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc58ca8840 00000020:00200000:2.0:1788572553.166183:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1788572553.166239:0:2395:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc58ca8840 00000100:00100000:2.0:1788572553.166268:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863922688:192.168.202.132@tcp:3:cat.0 00000100:00100000:1.0:1788572553.166589:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863922816:192.168.202.132@tcp:103:ldlm_bl.0 00000020:00010000:1.0:1788572553.167142:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3998 (ffff92fc6721ab48/0) at cl_lock_release():245 00000020:00010000:1.0:1788572553.167148:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3998 (ffff92fc6721ab48/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572553.167153:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70550530 (ffff92fc6721aab8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572553.167163:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572553.167179:0:111144:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:1.0:1788572553.167201:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3998 (ffff92fc6721ab48/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572553.167204:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70550530 (ffff92fc6721ab48/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572553.167211:0:111144:0:(vvp_io.c:533:vvp_io_advance()) advancing 2 bytes 00000080:00200000:1.0:1788572553.167224:0:111144:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040a:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572553.167228:0:111144:0:(file.c:2238:ll_file_io_generic()) subf: 1 io complete with rc: 0, result: 2, restart: 0 00000080:00200000:1.0:1788572553.167233:0:111144:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 2 00000080:00200008:1.0:1788572553.167318:0:111144:0:(file.c:2483:do_file_read_iter()) START file subf:[0x24000040a:0x2:0x0], ppos: 2, count: 4194304 00000080:00200000:1.0:1788572553.167331:0:111144:0:(file.c:2090:ll_file_io_generic()) subf: read ppos: 2, bytes: 4194304 00000080:00200000:1.0:1788572553.167335:0:111144:0:(file.c:1848:ll_io_set_mirror()) subf: desiginated mirror: 0 00000020:00200000:1.0:1788572553.167340:0:111144:0:(cl_io.c:214:cl_io_rw_init()) header@00000000db94255b[0x110, 2, [0x24000040a:0x2:0x0] hash] 00000020:00200000:1.0:1788572553.167342:0:111144:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2, 4194306) 0 0 00000080:00200000:1.0:1788572553.167346:0:111144:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040a:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572553.167361:0:111144:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 2 chunk: [2, 4194306) 4194306, 4194304 00020000:00200000:1.0:1788572553.167364:0:111144:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572553.167378:0:111144:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:1.0:1788572553.167390:0:111144:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572553.167394:0:111144:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4f30, osc lock ffff92fc47ff5430, flags 0x0 00010000:00010000:1.0:1788572553.167411:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167424:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167437:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167445:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167452:0:111144:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572553.167461:0:111144:0:(vvp_io.c:835:vvp_io_read_start()) subf: read [2, 4194306) 00000001:00010000:1.0:1788572553.167475:0:111144:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040a:0x2:0x0] 00000008:00010000:1.0:1788572553.167480:0:111144:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572553.167483:0:111144:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d42b0, osc lock ffff92fc47ff51d0, flags 0x400001002 00010000:00010000:1.0:1788572553.167490:0:111144:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167499:0:111144:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:1.0:1788572553.167507:0:111144:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572553.167510:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 7/2,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167518:0:111144:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 6/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:1.0:1788572553.167524:0:111144:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572553.167529:0:111144:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040a:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:1.0:1788572553.167532:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_release():245 00000020:00010000:1.0:1788572553.167534:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572553.167537:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d42b0 (ffff92fc6721aab8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572553.167540:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70477090 (ffff92fc6721ab48/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572553.167542:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d42b0 (ffff92fc6721ab48/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572553.167548:0:111144:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:2 00000020:00010000:1.0:1788572553.167554:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc464e3998 (ffff92fc6721ab48/0) at cl_lock_release():245 00000020:00010000:1.0:1788572553.167556:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464e3998 (ffff92fc6721ab48/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572553.167558:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4f30 (ffff92fc6721aab8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572553.167561:0:111144:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 4/1,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572553.167569:0:111144:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00010000:1.0:1788572553.167578:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464e3998 (ffff92fc6721ab48/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572553.167580:0:111144:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4f30 (ffff92fc6721ab48/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572553.167584:0:111144:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:1.0:1788572553.167589:0:111144:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040a:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572553.167593:0:111144:0:(file.c:2238:ll_file_io_generic()) subf: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:1.0:1788572553.167595:0:111144:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:1.0:1788572553.167701:0:111144:0:(file.c:479:ll_file_release()) START file subf:[0x24000040a:0x2:0x0](ffff92fc665eaa08), flags 100000 00000002:00100000:1.0:1788572553.167923:0:111144:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617f480 x1875449863922560/t4294976760(4294976760) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572569 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572553.167934:0:111144:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572553.167940:0:111144:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6728b480 x1875449863923072/t0(0) o35->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572553.167961:0:111144:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job cat:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111144:1875449863923072:192.168.202.132@tcp:35:cat.0 00000100:00100000:1.0:1788572553.168095:0:111144:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccf480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572553.171981:0:111144:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job cat:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111144:1875449863923072:192.168.202.132@tcp:35:cat.0 00000002:00100000:1.0:1788572553.172020:0:111144:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617f480 x1875449863922560/t4294976760(4294976760) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572569 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788572553.349153:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572554.379805:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572555.398207:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572555.781283:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923328:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.781382:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923584:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.781423:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1788572555.781429:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2394:1875449863923200:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.781455:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923456:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.781477:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923712:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.807994:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2394:1875449863923200:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.808019:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923456:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.808041:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923712:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.808050:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923328:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572555.808061:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2394:1875449863923584:192.168.202.132@tcp:400:kworker.0 00010000:00010000:3.0:1788572556.425841:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572557.452483:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572558.469056:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572559.503956:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572560.517915:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572560.901770:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863923968:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572560.901892:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863924224:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572560.901952:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572560.901959:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863923840:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572560.901986:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863924096:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572560.902009:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863924352:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572560.906398:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863923840:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572560.906416:0:2395:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc51d2a800: committing for last_committed 4294976761 gen 1 00000100:00100000:3.0:1788572560.906431:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863924096:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572560.906453:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863924352:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572560.906473:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863923968:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572560.906483:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863924224:192.168.202.132@tcp:400:kworker.0 00000080:00200000:3.0:1788572560.942708:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:3.0:1788572560.942768:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572560.942789:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572560.942806:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572560.942815:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572560.943419:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:3.0:1788572560.943424:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:3.0:1788572560.943433:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e0380d7fb58 00010000:00010000:3.0:1788572560.943437:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572560.943618:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572560.943636:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:3.0:1788572560.943645:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=8 00000080:00200008:3.0:1788572560.943654:0:111568:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572560.943659:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8),name=d59c.sanity-sec 00000002:00010000:3.0:1788572560.943675:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572560.943682:0:111568:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 30 res 8589935649/1 (0 0) 00010000:00010000:3.0:1788572560.944047:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.944056:0:111568:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.944068:0:111568:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572560.946138:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863924480:192.168.202.132@tcp:101:rm.0 00000100:00100000:3.0:1788572560.946275:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572560.954173:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863924480:192.168.202.132@tcp:101:rm.0 00010000:00010000:3.0:1788572560.954204:0:111568:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954223:0:111568:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954231:0:111568:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 4/1,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954246:0:111568:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954257:0:111568:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954272:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954281:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 5/2,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954288:0:111568:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 4/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572560.954299:0:111568:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733ad80 x1875449863924480/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572576 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572560.954312:0:111568:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954396:0:111568:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589935649/1 (0 0) 00000080:00200000:3.0:1788572560.954403:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572560.954428:0:111568:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572560.954434:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) for lock 0x58a604dcef352a98 00000080:00010000:3.0:1788572560.954518:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) 00000080:00010000:3.0:1788572560.954523:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a98 from it ffff9e0380d7fcb8 00010000:00010000:3.0:1788572560.954527:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.954537:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572560.954709:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:3.0:1788572560.955064:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572560.955076:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572560.955087:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572560.955093:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572560.955098:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:3.0:1788572560.955100:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:3.0:1788572560.955104:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e0380d7fbf8 00010000:00010000:3.0:1788572560.955107:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572560.955114:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572560.955123:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:3.0:1788572560.955129:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=266 00000080:00200000:3.0:1788572560.955135:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788572560.955142:0:111568:0:(file.c:1090:ll_file_open()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), flags 704400 00000002:00010000:3.0:1788572560.955157:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1788572560.955198:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.955206:0:111568:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572560.955218:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572560.955227:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6733ad80 x1875449863924736/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572560.955238:0:111568:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572560.955260:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863924736:192.168.202.132@tcp:101:rm.0 00000100:00100000:3.0:1788572560.955687:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572560.966405:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863924736:192.168.202.132@tcp:101:rm.0 00010000:00010000:3.0:1788572560.966431:0:111568:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966457:0:111568:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966466:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966474:0:111568:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966483:0:111568:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966489:0:111568:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204800 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:3.0:1788572560.966494:0:111568:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966506:0:111568:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966514:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966521:0:111568:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572560.966529:0:111568:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572560.966536:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204800/0x58a604dcef352a9f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572560.966556:0:111568:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc6733ad80 x1875449863924736/t4294983768(4294983768) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572560.966579:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6733ad80 x1875449863924736/t4294983768(4294983768) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572560.966596:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572560.966620:0:111568:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000002:00100000:3.0:1788572560.966644:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6733ad80 x1875449863924736/t4294983768(4294983768) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788572560.966714:0:111568:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572560.966809:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8),name=d59c.sanity-sec 00000002:00010000:3.0:1788572560.966843:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572560.966860:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572560.966878:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572560.966886:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572560.966891:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) 00000080:00010000:3.0:1788572560.966895:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) for lock 0x58a604dcef352a98 00000080:00010000:3.0:1788572560.966899:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a98 from it ffff9e0380d7fcd8 00010000:00010000:3.0:1788572560.966903:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572560.966912:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572560.967004:0:111568:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8) pos/size0/4096 32bit_api 0 00000002:00010000:3.0:1788572560.967399:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: readdir flags 0 00010000:00010000:3.0:1788572560.967407:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572560.967416:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788572560.968968:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863924864:192.168.202.132@tcp:37:rm.0 00000100:00100000:3.0:1788572560.969201:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572560.977789:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863924864:192.168.202.132@tcp:37:rm.0 00010000:00010000:3.0:1788572560.979516:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572560.979536:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1788572560.979764:0:111568:0:(file.c:479:ll_file_release()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), flags 704000 00000002:00100000:3.0:1788572560.979841:0:111568:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6733ad80 x1875449863924736/t4294983768(4294983768) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572560.979854:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572560.979860:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6733a300 x1875449863925120/t0(0) o35->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572560.979884:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925120:192.168.202.132@tcp:35:rm.0 00000100:00100000:2.0:1788572560.980493:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572560.985363:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925120:192.168.202.132@tcp:35:rm.0 00000002:00100000:2.0:1788572560.985405:0:111568:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6733ad80 x1875449863924736/t4294983768(4294983768) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572560.985503:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572560.985522:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572560.985543:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572560.985556:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572560.985564:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572560.985569:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572560.985573:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:2.0:1788572560.985581:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e0380d7fbf8 00010000:00010000:2.0:1788572560.985585:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572560.985593:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572560.985607:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:2.0:1788572560.985614:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=266 00000080:00200000:2.0:1788572560.985620:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572560.985628:0:111568:0:(file.c:1090:ll_file_open()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), flags 704400 00000002:00010000:2.0:1788572560.985646:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1788572560.985692:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572560.985700:0:111568:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572560.985710:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572560.985717:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617c380 x1875449863925376/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572560.985799:0:111568:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572560.985855:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925376:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572560.986020:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d9d000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572560.988818:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925376:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572560.988861:0:111568:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572560.988873:0:111568:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572560.988885:0:111568:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 4/1,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572560.988897:0:111568:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 4/1,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572560.988913:0:111568:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc4617c380 x1875449863925376/t4294983770(4294983770) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572560.988928:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c380 x1875449863925376/t4294983770(4294983770) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572560.988941:0:111568:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572560.988952:0:111568:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 16 res 8589935649/1 (0 0) 00000080:00200000:2.0:1788572560.988959:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572560.988980:0:111568:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000421:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572560.988985:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) for lock 0x58a604dcef352aa6 00000080:00010000:2.0:1788572560.988993:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352aa6 from it ffff9e0380d7fbd0 00010000:00010000:2.0:1788572560.988997:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 3/1,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572560.989008:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572560.989039:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617c380 x1875449863925376/t4294983770(4294983770) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572560.989108:0:111568:0:(file.c:6296:ll_getattr_dentry()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572560.989114:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8),name=d59c.sanity-sec 00000002:00010000:2.0:1788572560.989136:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000421:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572560.989146:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572560.989156:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572560.989164:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572560.989168:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) 00000080:00010000:2.0:1788572560.989172:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000421:0x1:0x0](ffff92fc665e00c8) for lock 0x58a604dcef352a98 00000080:00010000:2.0:1788572560.989176:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a98 from it ffff9e0380d7fcd8 00010000:00010000:2.0:1788572560.989180:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/1,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572560.989187:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572560.989281:0:111568:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8) pos/size0/4096 32bit_api 0 00000080:00200000:2.0:1788572560.989561:0:111568:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:2.0:1788572560.989586:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572560.989593:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=0 00000080:00200008:2.0:1788572560.989602:0:111568:0:(file.c:6296:ll_getattr_dentry()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572560.999070:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248),name=subdir 00000002:00010000:0.0:1788572560.999127:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572560.999144:0:111568:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 30 res 9663677450/1 (0 0) 00010000:00010000:0.0:1788572560.999204:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572560.999215:0:111568:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572560.999226:0:111568:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572560.999251:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925504:192.168.202.132@tcp:101:rm.0 00000100:00100000:0.0:1788572560.999342:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430b5000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572561.006057:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925504:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572561.006086:0:111568:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006115:0:111568:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006123:0:111568:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 4/1,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006136:0:111568:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006147:0:111568:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006159:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006168:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 5/2,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006175:0:111568:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 4/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572561.006187:0:111568:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6704b800 x1875449863925504/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/744 e 0 to 0 dl 1788572576 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572561.006207:0:111568:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006217:0:111568:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 4 res 9663677450/1 (0 0) 00000080:00200000:2.0:1788572561.006224:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572561.006262:0:111568:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572561.006271:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) for lock 0x58a604dcef352aad 00000080:00010000:2.0:1788572561.006276:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) 00000080:00010000:2.0:1788572561.006279:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352aad from it ffff9e0380d7fcb8 00010000:00010000:2.0:1788572561.006282:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.006291:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572561.006337:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.006348:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572561.006359:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572561.006366:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572561.006373:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572561.006429:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572561.006437:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=258 00000080:00200000:2.0:1788572561.007007:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248), inode mode 41ed mask 44 00000080:00200008:2.0:1788572561.007020:0:111568:0:(file.c:1090:ll_file_open()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), flags 704400 00000002:00010000:2.0:1788572561.007043:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788572561.007100:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.007108:0:111568:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572561.007118:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572561.007126:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d180 x1875449863925760/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/131864 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572561.007134:0:111568:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572561.007153:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925760:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572561.007413:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d9dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572561.016674:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925760:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572561.016699:0:111568:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016713:0:111568:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016889:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 4/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016899:0:111568:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016907:0:111568:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016914:0:111568:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366c00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572561.016919:0:111568:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016926:0:111568:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 4/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016933:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 5/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016940:0:111568:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572561.016946:0:111568:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 3/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.016953:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352ab4 lrc: 0/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572561.016978:0:111568:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617d180 x1875449863925760/t4294976762(4294976762) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/736 e 0 to 0 dl 1788572577 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572561.016991:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863925760/t4294976762(4294976762) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/736 e 0 to 0 dl 1788572577 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572561.017007:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572561.017043:0:111568:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572561.017062:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d180 x1875449863925760/t4294976762(4294976762) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/736 e 0 to 0 dl 1788572577 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572561.017119:0:111568:0:(file.c:6296:ll_getattr_dentry()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572561.017126:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248),name=subdir 00000002:00010000:2.0:1788572561.017163:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.017178:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572561.017190:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572561.017200:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.017206:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572561.017213:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572561.017220:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572561.017228:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572561.017243:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572561.017248:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) 00000080:00010000:2.0:1788572561.017252:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) for lock 0x58a604dcef352aad 00000080:00010000:2.0:1788572561.017256:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352aad from it ffff9e0380d7fcd8 00010000:00010000:2.0:1788572561.017260:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572561.017267:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572561.017280:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.017286:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572561.017293:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572561.017300:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572561.017307:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572561.017333:0:111568:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248) pos/size0/4096 32bit_api 0 00010000:00010000:2.0:1788572561.017412:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572561.017551:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572561.017560:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572561.017569:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000100:00100000:2.0:1788572561.018210:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925888:192.168.202.132@tcp:37:rm.0 00000100:00100000:2.0:1788572561.018553:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d9dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572561.048833:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863925888:192.168.202.132@tcp:37:rm.0 00010000:00010000:2.0:1788572561.049499:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572561.049515:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200008:2.0:1788572561.049612:0:111568:0:(file.c:479:ll_file_release()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), flags 704000 00000002:00100000:2.0:1788572561.049680:0:111568:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617d180 x1875449863925760/t4294976762(4294976762) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/736 e 0 to 0 dl 1788572577 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572561.049692:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572561.049697:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ed80 x1875449863926144/t0(0) o35->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572561.049751:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926144:192.168.202.132@tcp:35:rm.0 00000100:00100000:2.0:1788572561.049845:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d9dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572561.056293:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926144:192.168.202.132@tcp:35:rm.0 00000002:00100000:2.0:1788572561.056336:0:111568:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617d180 x1875449863925760/t4294976762(4294976762) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/736 e 0 to 0 dl 1788572577 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572561.056415:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572561.056424:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=258 00000080:00200000:2.0:1788572561.056431:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248), inode mode 41ed mask 44 00000080:00200008:2.0:1788572561.056440:0:111568:0:(file.c:1090:ll_file_open()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), flags 704400 00000002:00010000:2.0:1788572561.056464:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1788572561.056522:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.056531:0:111568:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572561.056552:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572561.056768:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617ed80 x1875449863926400/t0(0) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/131864 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572561.056779:0:111568:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 3/1,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111568 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572561.056798:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926400:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572561.058096:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc44d9dd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572561.068145:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926400:192.168.202.132@tcp:101:rm.0 00010000:00010000:2.0:1788572561.068188:0:111568:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.068199:0:111568:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 4/1,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.068212:0:111568:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 4/1,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.068224:0:111568:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 4/1,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572561.068245:0:111568:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff92fc4617ed80 x1875449863926400/t4294976764(4294976764) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/736 e 0 to 0 dl 1788572577 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572561.068258:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ed80 x1875449863926400/t4294976764(4294976764) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 656/736 e 0 to 0 dl 1788572577 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572561.068273:0:111568:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 3/1,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.068283:0:111568:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02c00 type 13 mode 16 res 9663677450/1 (0 0) 00000080:00200000:2.0:1788572561.068289:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572561.068325:0:111568:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040a:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572561.068333:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) for lock 0x58a604dcef352abb 00000080:00010000:2.0:1788572561.068341:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352abb from it ffff9e0380d7fbd0 00010000:00010000:2.0:1788572561.068344:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 3/1,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572561.068354:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 2/0,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572561.068380:0:111568:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617ed80 x1875449863926400/t4294976764(4294976764) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/736 e 0 to 0 dl 1788572577 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572561.068441:0:111568:0:(file.c:6296:ll_getattr_dentry()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572561.068449:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248),name=subdir 00000002:00010000:2.0:1788572561.068498:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040a:0x1:0x0]) in obj [0x200000421:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.068510:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572561.068527:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00010000:2.0:1788572561.068537:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.068551:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572561.068566:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572561.068575:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572561.068583:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572561.068593:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000421:0x1:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788572561.068598:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) 00000080:00010000:2.0:1788572561.068601:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040a:0x1:0x0](ffff92fc665eb248) for lock 0x58a604dcef352aad 00000080:00010000:2.0:1788572561.068605:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352aad from it ffff9e0380d7fcd8 00010000:00010000:2.0:1788572561.068609:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 3/1,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572561.068616:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:2.0:1788572561.068627:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572561.068634:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572561.068641:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572561.068649:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572561.068656:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572561.068684:0:111568:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff92fc51d2a800) subdir proj=0 00000080:00000004:2.0:1788572561.068688:0:111568:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc51d2a800: age 4306, max_age 4315 00800000:00000004:2.0:1788572561.068697:0:111568:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc51d2a800: age 4306, max_age 4315 00000100:00100000:2.0:1788572561.068784:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926528:192.168.202.132@tcp:41:rm.0 00000100:00100000:2.0:1788572561.069136:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60b34300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572561.075277:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926528:192.168.202.132@tcp:41:rm.0 00800000:00000004:3.0:1788572561.075306:0:111568:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc51d2a800: update ffff92fc70030458 cache blocks 1802684/1916558 objects 523525/524093 00000080:00000004:3.0:1788572561.075314:0:111568:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc51d2a800: update ffff92fc700337d8 cache blocks 1802684/1916558 objects 523525/524093 00000080:00000004:3.0:1788572561.075318:0:111568:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 1802684/1916558 objects 523525/524093 00000080:00200000:3.0:1788572561.075350:0:111568:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248) pos/size0/4096 32bit_api 0 00010000:00010000:3.0:1788572561.075377:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00010000:3.0:1788572561.075643:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240000400:0x5:0x0]) in obj [0x240000400:0x5:0x0], intent: readdir flags 0 00010000:00010000:3.0:1788572561.075657:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572561.075668:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000002:00200000:3.0:1788572561.075688:0:111568:0:(mdc_request.c:1174:mdc_page_locate()) offset fffffffffffffffe [0x0 0xfffffffffffffffe], hash 0x0 00010000:00010000:3.0:1788572561.075692:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 3/1,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572561.075700:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572561.075948:0:111568:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:3.0:1788572561.075970:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248), inode mode 41ed mask 1 00000080:00200000:3.0:1788572561.075980:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subf, flags=0 00000080:00200000:3.0:1788572561.075986:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248), inode mode 41ed mask 3 00000080:00200000:3.0:1788572561.075991:0:111568:0:(namei.c:2351:ll_unlink()) VFS Op:name=subf, dir=[0x24000040a:0x1:0x0](ffff92fc665eb248) 00010000:00010000:3.0:1788572561.076054:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572561.076110:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572561.076121:0:111568:0:(vvp_object.c:117:vvp_conf_set()) [0x24000040a:0x2:0x0]: losing layout lock 00010000:00010000:3.0:1788572561.076151:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572561.076159:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 1/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572561.076164:0:111568:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572561.076167:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3000/0x58a604dcef352a83 lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792f34 expref: -99 pid: 110779 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572561.076187:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2800/0x58a604dcef352a7c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f18 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000020:00100000:3.0:1788572561.076203:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572561.076211:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6733a300 x1875449863926784/t0(0) o36->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 504/632 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572561.076231:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926784:192.168.202.132@tcp:36:rm.0 00000100:00100000:3.0:1788572561.076359:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572561.092165:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863926784:192.168.202.132@tcp:36:rm.0 00000080:00200000:3.0:1788572561.092237:0:111568:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040a:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572561.092252:0:111568:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572561.092328:0:111568:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040a:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572561.092390:0:111568:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040a:0x2:0x0](ffff92fc665eaa08) 00000008:00010000:3.0:1788572561.092432:0:111568:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 2 blocks 8 [cma]time: 1788572547 1788572547 0 by oinfo size 2 blocks 8 [cma]time 1788572547 1788572547 1788572553 ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 1/0,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000080:00200008:3.0:1788572561.092507:0:111568:0:(file.c:479:ll_file_release()) START file subdir:[0x24000040a:0x1:0x0](ffff92fc665eb248), flags 704000 00010000:00010000:3.0:1788572561.092533:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 2/0,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572561.092691:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572561.092701:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=0 00000080:00200000:3.0:1788572561.092707:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788572561.092713:0:111568:0:(namei.c:2261:ll_rmdir()) VFS Op:name=subdir, dir=[0x200000421:0x1:0x0](ffff92fc665e00c8) 00010000:00010000:3.0:1788572561.092968:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572561.093023:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572561.093031:0:111568:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572561.093109:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863926912:192.168.202.132@tcp:103:rm.0 00010000:00010000:3.0:1788572561.093484:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70367400/0x58a604dcef352a6e lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792efc expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572561.093525:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 2/0,0 mode: CR/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:3.0:1788572561.093574:0:111568:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617ed80 x1875449863926400/t4294976764(4294976764) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/736 e 0 to 0 dl 1788572577 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572561.093586:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572561.093591:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6733b100 x1875449863927168/t0(0) o35->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572561.093608:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863927168:192.168.202.132@tcp:35:rm.0 00000100:00100000:3.0:1788572561.093650:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572561.103953:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863927168:192.168.202.132@tcp:35:rm.0 00000002:00100000:3.0:1788572561.103994:0:111568:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617ed80 x1875449863926400/t4294976764(4294976764) o101->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 592/736 e 0 to 0 dl 1788572577 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788572561.104050:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572561.104068:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 2/0,0 mode: PR/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572561.104108:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 1/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572561.104116:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572561.104122:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 1/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572561.104127:0:111568:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788572561.104130:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366c00/0x58a604dcef352abb lrc: 0/0,0 mode: --/CR res: [0x24000040a:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792fab expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572561.104143:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc51bc3200/0x58a604dcef352aad lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f8f expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572561.104151:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc51d2a800 lock: ffff92fc70366e00/0x58a604dcef352a75 lrc: 0/0,0 mode: --/PR res: [0x24000040a:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f0a expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572561.104172:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572561.104178:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6733b100 x1875449863927424/t0(0) o36->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 512/632 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572561.104199:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863927424:192.168.202.132@tcp:36:rm.0 00000100:00100000:3.0:1788572561.104349:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572561.106709:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863926912:192.168.202.132@tcp:103:rm.0 00000100:00100000:1.0:1788572561.107453:0:2386:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.132@tcp (source: 12345-192.168.202.132@tcp) 00000100:00100000:0.0:1788572561.107537:0:111060:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875449881897088 00000100:00100000:0.0:1788572561.107556:0:111060:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875449881897088 00000100:00100000:0.0:1788572561.107606:0:111060:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.132@tcp, seq: 3 00000100:00100000:0.0:1788572561.107616:0:111060:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:130682:x1875449881897088:12345-192.168.202.132@tcp:104: 00010000:00010000:0.0:1788572561.107635:0:111060:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572561.107712:0:111060:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788572561.107738:0:111060:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff92fc67288380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:969e09c0-9457-45fe-bdd1-1282a4dee8fd+5:130682:x1875449881897088:12345-192.168.202.132@tcp:104: Request processed in 106us (271us total) trans 0 rc 0/0 00000100:00100000:0.0:1788572561.107746:0:111060:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.132@tcp, seq: 3 00010000:00010000:2.0:1788572561.108472:0:111129:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572561.108482:0:111129:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108491:0:111129:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70204200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572561.108496:0:111129:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108509:0:111129:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108517:0:111129:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 3/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108552:0:111129:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108607:0:111129:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 2/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108614:0:111129:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572561.108643:0:111129:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108651:0:111129:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70204200/0x58a604dcef352a98 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xc9a239a221792f5e expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572561.108666:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572561.108668:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572561.108670:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572561.108677:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572561.108693:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863927552:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572561.117336:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2395:1875449863927552:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572561.124484:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863927424:192.168.202.132@tcp:36:rm.0 00000080:00200000:3.0:1788572561.124531:0:111568:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040a:0x1:0x0](ffff92fc665eb248) 00000080:00200008:3.0:1788572561.124635:0:111568:0:(file.c:479:ll_file_release()) START file d59c.sanity-sec:[0x200000421:0x1:0x0](ffff92fc665e00c8), flags 704000 00010000:00010000:3.0:1788572561.124674:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572561.124803:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:3.0:1788572561.124840:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572561.124853:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572561.124862:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572561.124869:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572561.124875:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:3.0:1788572561.124879:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:3.0:1788572561.124887:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e0380d7fc10 00010000:00010000:3.0:1788572561.124891:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572561.124899:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572561.124914:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 1 00000080:00200000:3.0:1788572561.124923:0:111568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d59c.sanity-sec, flags=0 00000080:00200000:3.0:1788572561.124929:0:111568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:3.0:1788572561.124935:0:111568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572561.124946:0:111568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572561.124959:0:111568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572561.124965:0:111568:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572561.124969:0:111568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:3.0:1788572561.124971:0:111568:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352a60 00000080:00010000:3.0:1788572561.124974:0:111568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352a60 from it ffff9e0380d7fd88 00010000:00010000:3.0:1788572561.124977:0:111568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572561.124984:0:111568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572561.124992:0:111568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448), inode mode 41ed mask 3 00000080:00200000:3.0:1788572561.124997:0:111568:0:(namei.c:2261:ll_rmdir()) VFS Op:name=d59c.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ef448) 00010000:00010000:3.0:1788572561.125015:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572561.125035:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 2/0,0 mode: CR/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:3.0:1788572561.125076:0:111568:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617c380 x1875449863925376/t4294983770(4294983770) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572561.125087:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572561.125092:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67339f80 x1875449863927808/t0(0) o35->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572561.125118:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863927808:192.168.202.132@tcp:35:rm.0 00000100:00100000:3.0:1788572561.125183:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572561.129543:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863927808:192.168.202.132@tcp:35:rm.0 00000002:00100000:3.0:1788572561.129584:0:111568:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617c380 x1875449863925376/t4294983770(4294983770) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 760/608 e 0 to 0 dl 1788572576 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788572561.129643:0:111568:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 2/0,0 mode: PR/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572561.129685:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 1/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572561.129694:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 1/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572561.129702:0:111568:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572561.129709:0:111568:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788572561.129712:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366400/0x58a604dcef352aa6 lrc: 0/0,0 mode: --/CR res: [0x200000421:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792f7a expref: -99 pid: 111568 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572561.129905:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc70366200/0x58a604dcef352a67 lrc: 0/0,0 mode: --/PR res: [0x200000421:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ee0 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572561.129921:0:111568:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc51bc2200/0x58a604dcef352a60 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221792ec4 expref: -99 pid: 110779 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000020:00100000:3.0:1788572561.129947:0:111568:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc51d2a800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572561.129954:0:111568:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67339f80 x1875449863928064/t0(0) o36->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 504/736 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572561.129977:0:111568:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863928064:192.168.202.132@tcp:36:rm.0 00000100:00100000:3.0:1788572561.130100:0:111568:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e8d900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572561.147049:0:111568:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job rm:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111568:1875449863928064:192.168.202.132@tcp:36:rm.0 00000080:00200000:3.0:1788572561.147099:0:111568:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000421:0x1:0x0](ffff92fc665e00c8) 00010000:00010000:3.0:1788572561.544426:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572561.726201:0:111585:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665ef448), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572561.726213:0:111585:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:1.0:1788572561.726251:0:111585:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572561.726321:0:111585:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111585 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572561.726349:0:111585:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111585 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572561.726368:0:111585:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111585 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572561.726391:0:111585:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job lsof:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111585:1875449863928192:192.168.202.132@tcp:101:lsof.0 00000100:00100000:1.0:1788572561.726466:0:111585:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45182540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572561.729001:0:111585:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job lsof:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111585:1875449863928192:192.168.202.132@tcp:101:lsof.0 00010000:00010000:3.0:1788572561.729034:0:111585:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729042:0:111585:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729051:0:111585:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729059:0:111585:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729068:0:111585:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729073:0:111585:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729079:0:111585:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572561.729087:0:111585:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728a680 x1875449863928192/t0(0) o101->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572561.729096:0:111585:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729107:0:111585:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc510de800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572561.729113:0:111585:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572561.729139:0:111585:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572561.729143:0:111585:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352ac2 00000080:00010000:3.0:1788572561.729147:0:111585:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:3.0:1788572561.729149:0:111585:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ac2 from it ffff9e0381477cb8 00010000:00010000:3.0:1788572561.729151:0:111585:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572561.729158:0:111585:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572562.578110:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572563.593986:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572564.156386:0:111593:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665ef448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572564.156399:0:111593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572564.156425:0:111593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572564.156444:0:111593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572564.156461:0:111593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572564.156470:0:111593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572564.156478:0:111593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572564.156481:0:111593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352ac2 00000080:00010000:2.0:1788572564.156489:0:111593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ac2 from it ffff9e0381d4bcb8 00010000:00010000:2.0:1788572564.156493:0:111593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572564.156502:0:111593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788572564.156642:0:111593:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665ef448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572564.156647:0:111593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448),name=/ 00000002:00010000:2.0:1788572564.156654:0:111593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572564.156662:0:111593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572564.156671:0:111593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572564.156684:0:111593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572564.156688:0:111593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00010000:2.0:1788572564.156691:0:111593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ef448) for lock 0x58a604dcef352ac2 00000080:00010000:2.0:1788572564.156694:0:111593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ac2 from it ffff9e0381d4bcb8 00010000:00010000:2.0:1788572564.156718:0:111593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572564.156836:0:111593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572564.173051:0:111593:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ef448) 00000080:00200000:2.0:1788572564.173085:0:111593:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000400:0x5:0x0](ffff92fc665ee3c8) 00000080:00200000:2.0:1788572564.173095:0:111593:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc51d2a800 (sb ffff92fc51d2a800) 00010000:00010000:2.0:1788572564.173135:0:111593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173147:0:111593:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173155:0:111593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173164:0:111593:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173183:0:111593:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788572564.173188:0:111593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173193:0:111593:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173197:0:111593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173201:0:111593:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173215:0:111593:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:2.0:1788572564.173219:0:111593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173223:0:111593:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173226:0:111593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173229:0:111593:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173238:0:111593:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:2.0:1788572564.173241:0:111593:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:2.0:1788572564.173768:0:111593:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc49909000 active: 1 00010000:00010000:3.0:1788572564.173771:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00020000:01000000:2.0:1788572564.173774:0:111593:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc6722f000 active: 1 00020000:01000000:2.0:1788572564.173776:0:111593:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00010000:00010000:3.0:1788572564.173778:0:111069:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00020000:01000000:2.0:1788572564.173781:0:111593:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc51d2a800 00020000:01000000:2.0:1788572564.173783:0:111593:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc51d2a800: disconnecting target lustre-OST0001-osc-ffff92fc51d2a800 00010000:00010000:3.0:1788572564.173786:0:111069:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205400 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:3.0:1788572564.173790:0:111069:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572564.173794:0:111593:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-OST0001_UUID 00010000:00010000:3.0:1788572564.173798:0:111069:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.173805:0:111069:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.173810:0:111069:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:3.0:1788572564.173812:0:111069:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:3.0:1788572564.173814:0:111069:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:2.0:1788572564.173833:0:111593:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572564.173855:0:111069:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173857:0:111593:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572564.173862:0:111069:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572564.173864:0:111593:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572564.173876:0:111593:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53da9b40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572564.173881:0:111069:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572564.173886:0:111593:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job umount:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111593:1875449863928448:192.168.202.132@tcp:103:umount.0 00010000:00010000:2.0:1788572564.173896:0:111593:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572564.173900:0:111069:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352972 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c4e expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572564.173902:0:111593:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572564.173905:0:111593:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff92fc51d2a800 lock: 00000000b75b0a0d/0x58a604dcef352a8a lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x21a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221792f3b expref: -99 pid: 110779 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572564.173910:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788572564.173914:0:111069:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.173919:0:111069:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366000 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:3.0:1788572564.173922:0:111069:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.173928:0:111069:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.173935:0:111069:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.173941:0:111069:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:3.0:1788572564.173943:0:111069:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:3.0:1788572564.173944:0:111069:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:3.0:1788572564.173955:0:111069:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.173961:0:111069:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572564.173967:0:111069:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.173974:0:111069:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366000/0x58a604dcef3529a3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c55 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.173985:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572564.173987:0:111069:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572564.173988:0:111593:0:(import.c:65:import_set_state_nolock()) ffff92fc492a5000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00010000:00010000:3.0:1788572564.173992:0:111069:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367600 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:3.0:1788572564.173995:0:111069:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.174001:0:111069:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572564.174007:0:111069:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.174016:0:111069:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:3.0:1788572564.174018:0:111069:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:3.0:1788572564.174020:0:111069:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:3.0:1788572564.174031:0:111069:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.174038:0:111069:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572564.174043:0:111069:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.174048:0:111069:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70367600/0x58a604dcef3529aa lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c5c expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.174069:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572564.174071:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572564.174073:0:111069:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572564.174089:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863928320:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572564.174202:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863928832:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572564.174230:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572564.174236:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863928576:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572564.174273:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863928704:192.168.202.132@tcp:9:umount.0 00010000:00010000:0.0:1788572564.174283:0:111070:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572564.175463:0:111129:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572564.180329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc492a5000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572564.180365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863928704:192.168.202.132@tcp:9:umount.0 00000100:00100000:2.0:1788572564.180439:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863928576:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:0.0:1788572564.180451:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572564.180456:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc51d2a800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572564.180461:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572564.180598:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3529d4 00000020:00000080:0.0:1788572564.180616:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc6722f000/LOV_OSC_UUID 00000020:01000000:0.0:1788572564.180674:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc51d2a800 (flags='') 00000020:00000080:0.0:1788572564.180681:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572564.180691:0:111593:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc51d2a800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572564.180704:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc51cdb380) has no exports 00000020:00000080:3.0:1788572564.181146:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc6722f000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc51d2a800 00000008:01000000:0.0:1788572564.181207:0:111593:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc51d2a800: client import never connected 00000100:00080000:0.0:1788572564.181211:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572564.181214:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc51d2a800: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572564.181217:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572564.181424:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572564.181431:0:111593:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00020000:01000000:0.0:1788572564.181435:0:111593:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc51d2a800 00020000:01000000:0.0:1788572564.181437:0:111593:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc51d2a800: disconnecting target lustre-OST0000-osc-ffff92fc51d2a800 00000100:00080000:0.0:1788572564.181449:0:111593:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-OST0000_UUID 00000100:00080000:0.0:1788572564.181549:0:111593:0:(import.c:65:import_set_state_nolock()) ffff92fc492a3800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572564.181638:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863928960:192.168.202.132@tcp:9:umount.0 00000100:00100000:2.0:1788572564.181910:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863928320:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572564.181930:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2395:1875449863928832:192.168.202.132@tcp:103:ldlm_bl.0 00000020:00000080:3.0:1788572564.183235:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc492a5000 for lustre-OST0001-osc-ffff92fc51d2a800 00000020:00000080:3.0:1788572564.183249:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc492a7800/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:3.0:1788572564.183252:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a7800/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-OST0001-osc-ffff92fc51d2a800 00000020:01000000:3.0:1788572564.183261:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc51d2a800 (969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000100:00080000:3.0:1788572564.197051:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc492a3800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572564.197077:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863928960:192.168.202.132@tcp:9:umount.0 00000100:00080000:0.0:1788572564.197208:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572564.197212:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc51d2a800: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572564.197217:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572564.197351:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3529cd 00000020:00000080:0.0:1788572564.197373:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc49909000/LOV_OSC_UUID 00000020:01000000:0.0:1788572564.197435:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc51d2a800 (flags='') 00000020:00000080:0.0:1788572564.197442:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572564.197453:0:111593:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc51d2a800: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572564.197458:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc51cd99c0) has no exports 00000020:00000080:2.0:1788572564.197863:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc49909000/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc51d2a800 00000008:01000000:0.0:1788572564.197919:0:111593:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc51d2a800: client import never connected 00000100:00080000:0.0:1788572564.197923:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572564.197925:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc51d2a800: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572564.197928:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572564.198082:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572564.198088:0:111593:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:00000080:0.0:1788572564.198092:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3529c6 00000020:00000080:0.0:1788572564.198097:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc42b6f000/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000100:00080000:0.0:1788572564.198159:0:111593:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-MDT0000_UUID 00010000:00010000:0.0:1788572564.198278:0:111593:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572564.198332:0:111593:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572564.198340:0:111593:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572564.198356:0:111593:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc5105e780 going to sleep for 0 seconds 00000100:00100000:0.0:1788572564.198365:0:111593:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job umount:969e09c0-9457-45fe-bdd1-1282a4dee8fd:111593:1875449863929088:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572564.198375:0:111593:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572564.198380:0:111593:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572564.198384:0:111593:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc51d2a800 lock: ffff92fc702f5000/0x58a604dcef352ac2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793029 expref: -99 pid: 111585 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00000080:2.0:1788572564.198845:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc492a3800 for lustre-OST0000-osc-ffff92fc51d2a800 00000020:00000080:2.0:1788572564.198854:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc45368800/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:2.0:1788572564.198856:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc45368800/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-OST0000-osc-ffff92fc51d2a800 00000020:01000000:2.0:1788572564.198862:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc51d2a800 (969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:00000080:2.0:1788572564.199109:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc42b6f000/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-clilov-ffff92fc51d2a800 00000100:00080000:0.0:1788572564.204077:0:111593:0:(import.c:65:import_set_state_nolock()) ffff92fc46304800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572564.207895:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863929216:192.168.202.132@tcp:39:umount.0 00000100:00080000:3.0:1788572564.211360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc46304800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572564.211395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863929216:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572564.211447:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572564.211452:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc51d2a800: committing for last_committed 4294983765 gen 2 00000100:00100000:0.0:1788572564.211462:0:111593:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67339f80 x1875449863928064/t4294983773(4294983773) o36->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 504/456 e 0 to 0 dl 1788572577 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572564.211486:0:111593:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6704bb80 x1875449863929216/t4294983774(4294983774) o39->lustre-MDT0000-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572574 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572564.211499:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572564.213427:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3529b8 00000020:00000080:0.0:1788572564.213443:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc492a1000/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000100:00080000:0.0:1788572564.213539:0:111593:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 969e09c0-9457-45fe-bdd1-1282a4dee8fd->lustre-MDT0001_UUID 00000020:00000080:1.0:1788572564.214976:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a1000/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-MDT0000-mdc-ffff92fc51d2a800 00000100:00080000:0.0:1788572564.215359:0:111593:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572564.215417:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863929344:192.168.202.132@tcp:39:umount.0 00000100:00100000:3.0:1788572564.226398:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc51d2a800: committing for last_committed 4294976767 gen 1 00000100:00100000:3.0:1788572564.226432:0:2392:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff92fc6704bb80 x1875449863929344/t4294976769(4294976769) o39->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572574 ref 2 fl Rpc:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572564.226446:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572564.226469:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:969e09c0-9457-45fe-bdd1-1282a4dee8fd:2392:1875449863929344:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572564.226542:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788572564.226546:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc51d2a800: committing for last_committed 4294976767 gen 2 00000100:00100000:0.0:1788572564.226554:0:111593:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6704bb80 x1875449863929344/t4294976769(4294976769) o39->lustre-MDT0001-mdc-ffff92fc51d2a800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572574 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572564.226570:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572564.228451:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3529bf 00000020:00000080:0.0:1788572564.228466:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc492a0000/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:0.0:1788572564.228510:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef3529b1 00000020:00000080:0.0:1788572564.228514:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc492a6800/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:01000000:0.0:1788572564.228523:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc51d2a800 (flags='') 00000020:00000080:0.0:1788572564.228531:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572564.228538:0:111593:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc51d2a800: forcing exports to disconnect: 0/1 00000020:00080000:0.0:1788572564.228542:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc70034d40) has no exports 00000020:00000080:0.0:1788572564.228619:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572564.228626:0:111593:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:00000080:0.0:1788572564.228629:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc4a3b6800/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:0.0:1788572564.228632:0:111593:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4a3b6800/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-clilov-ffff92fc51d2a800 00000020:01000000:0.0:1788572564.228637:0:111593:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc51d2a800 (969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572564.228864:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc51d2a800 (flags='') 00000020:00000080:0.0:1788572564.228868:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572564.228871:0:111593:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc51d2a800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572564.228873:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc70033380) has no exports 00000020:00000080:0.0:1788572564.228917:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572564.228921:0:111593:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:0.0:1788572564.228924:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc51d2a800 (flags='') 00000020:00000080:0.0:1788572564.228926:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572564.228929:0:111593:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc51d2a800: forcing exports to disconnect: 0/2 00000020:00080000:0.0:1788572564.228931:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc70030000) has no exports 00000020:00000080:1.0:1788572564.231503:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a0000/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-MDT0001-mdc-ffff92fc51d2a800 00000020:00000080:1.0:1788572564.231519:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a6800/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-clilmv-ffff92fc51d2a800 00000020:00000080:1.0:1788572564.231525:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc4a3b0800/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:1.0:1788572564.231528:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4a3b0800/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-clilmv-ffff92fc51d2a800 00000020:01000000:1.0:1788572564.231531:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc51d2a800 (969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000008:01000000:0.0:1788572564.233857:0:111593:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc51d2a800: client import never connected 00000100:00080000:0.0:1788572564.233866:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572564.233871:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc51d2a800: committing for last_committed 4294983765 gen 3 00000100:00100000:0.0:1788572564.233882:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572564.248487:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572564.248502:0:111593:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000020:01000000:1.0:1788572564.248509:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc51d2a800 (flags='') 00000020:00000080:1.0:1788572564.248513:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572564.248516:0:111593:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc51d2a800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572564.248520:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc700319c0) has no exports 00000020:00000080:0.0:1788572564.250060:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc46304800 for lustre-MDT0000-mdc-ffff92fc51d2a800 00000020:00000080:0.0:1788572564.250077:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc4a3b7000/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:0.0:1788572564.250081:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4a3b7000/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-MDT0000-mdc-ffff92fc51d2a800 00000020:01000000:0.0:1788572564.250096:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc51d2a800 (969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000008:01000000:1.0:1788572564.251583:0:111593:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc51d2a800: client import never connected 00000100:00080000:1.0:1788572564.251591:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572564.251597:0:111593:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc51d2a800: committing for last_committed 4294976767 gen 3 00000100:00100000:1.0:1788572564.251608:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572564.264361:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572564.265305:0:111593:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc51d2a800 (uuid 969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000080:02000400:1.0:1788572564.265313:0:111593:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:1.0:1788572564.265316:0:111593:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:1.0:1788572564.265464:0:111593:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc51d2a800 00000100:00080000:1.0:1788572564.265473:0:111593:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1->MGS 00010000:00010000:1.0:1788572564.265490:0:111593:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572564.265503:0:111593:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572564.265511:0:111593:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:1.0:1788572564.265513:0:111593:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:1.0:1788572564.265517:0:111593:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:1.0:1788572564.265569:0:111593:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572564.265577:0:111593:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572564.265597:0:111593:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988840 going to sleep for 0 seconds 00000100:00100000:1.0:1788572564.265612:0:111593:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job umount:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:111593:1875449863929472:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572564.265623:0:111593:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572564.265632:0:111593:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572564.265636:0:111593:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3400/0x58a604dcef35296b lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221792c47 expref: -99 pid: 111058 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572564.265669:0:111593:0:(import.c:65:import_set_state_nolock()) ffff92fc42c08000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572564.266327:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2392:1875449863929600:192.168.202.132@tcp:251:umount.0 00000020:00000080:0.0:1788572564.267524:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4536c800 for lustre-MDT0001-mdc-ffff92fc51d2a800 00000020:00000080:0.0:1788572564.267536:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc53e08000/969e09c0-9457-45fe-bdd1-1282a4dee8fd 00000020:00000080:0.0:1788572564.267540:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc53e08000/969e09c0-9457-45fe-bdd1-1282a4dee8fd for lustre-MDT0001-mdc-ffff92fc51d2a800 00000020:01000000:0.0:1788572564.267547:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc51d2a800 (969e09c0-9457-45fe-bdd1-1282a4dee8fd) 00000100:00080000:3.0:1788572564.279232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c08000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:3.0:1788572564.279245:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572564.279249:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572564.279250:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:3.0:1788572564.279379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1:2392:1875449863929600:192.168.202.132@tcp:251:umount.0 00000100:00080000:2.0:1788572564.279500:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572564.279505:0:111593:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572564.279507:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572564.279510:0:111593:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572564.279520:0:111593:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352964 00000020:00000080:2.0:1788572564.279528:0:111593:0:(genops.c:885:class_export_put()) final put ffff92fc42c0b800/71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1 00000020:01000000:2.0:1788572564.279547:0:111593:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:2.0:1788572564.279553:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572564.279560:0:111593:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572564.279564:0:111593:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc49934d40) has no exports 00000020:00000080:0.0:1788572564.279633:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc42c0b800/71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1 for MGC192.168.202.132@tcp 10000000:01000000:0.0:1788572564.279684:0:111072:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788572564.280077:0:111593:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:2.0:1788572564.280080:0:111593:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572564.280083:0:111593:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572564.280085:0:111593:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572564.280087:0:111593:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572564.280178:0:111593:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572564.280184:0:111593:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1) 00000020:01000004:2.0:1788572564.280200:0:111593:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc51d2a800 1 00000020:01000004:2.0:1788572564.280203:0:111593:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc51d28000 00000020:00000080:3.0:1788572564.280502:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc42c08000 for MGC192.168.202.132@tcp 00000020:00000080:3.0:1788572564.280509:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc51d2c000/71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1 00000020:00000080:3.0:1788572564.280513:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2c000/71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1 for MGC192.168.202.132@tcp 00000020:01000000:3.0:1788572564.280523:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (71c23ab1-5cbc-44d2-8a42-1a40a43aaaa1) 00010000:00010000:3.0:1788572564.280804:0:111069:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:3.0:1788572564.280941:0:111070:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:3.0:1788572564.281046:0:111129:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00000100:00100000:0.0:1788572564.281566:0:111060:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc49a42e00:111060] 1 exiting: rc = 0 00000100:00100000:1.0:1788572564.281624:0:111059:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc53dfe600:111059] 0 exiting: rc = 1 00000100:00100000:3.0:1788572564.285557:0:111062:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff92fc53f75a00:111062] 1 exiting: rc = 0 00000100:00100000:2.0:1788572564.291902:0:111061:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc49a43800:111061] 0 exiting: rc = 1 00000100:00100000:3.0:1788572564.292277:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:3.0:1788572564.292290:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572564.295089:0:111063:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc484ce600:111063] 0 exiting: rc = 1 00000100:00100000:1.0:1788572564.295179:0:111064:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc484ce400:111064] 1 exiting: rc = 0 00000100:00100000:0.0:1788572564.295528:0:111065:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc49d70200:111065] 2 exiting: rc = 0 00000100:00100000:2.0:1788572564.296874:0:111066:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc484cf800:111066] 0 exiting: rc = 1 00000100:00100000:2.0:1788572564.296953:0:111067:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc60ab3400:111067] 1 exiting: rc = 0 00000100:00100000:2.0:1788572564.297002:0:111068:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc470f3e00:111068] 2 exiting: rc = 0 00000100:00100000:2.0:1788572564.297325:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297333:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297339:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297344:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297349:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297354:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297360:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297431:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297440:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297446:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297451:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297457:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297462:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297467:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297473:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297477:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297482:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297488:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297494:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297498:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297503:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297509:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297514:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297519:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297524:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297529:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297534:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297542:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297546:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297551:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297556:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297561:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297567:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297572:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297577:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297582:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297587:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297592:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297598:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297604:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297609:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297614:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297619:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297625:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297630:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297635:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297641:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297645:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297650:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297656:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297661:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297666:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297671:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297677:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297682:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297686:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297701:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297707:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297712:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297717:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297744:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297750:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297755:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297761:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297767:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297772:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297777:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297782:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297786:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297792:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297797:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297802:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297807:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297813:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297818:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297822:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297827:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297832:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297837:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297842:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297846:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297851:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297857:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297862:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297866:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297871:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297876:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297881:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297886:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297893:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297898:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297904:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297909:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297914:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297918:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297922:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297927:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297933:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297938:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297943:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297949:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297953:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297959:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297964:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297969:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297974:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297979:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297984:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297990:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.297995:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298000:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298005:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298010:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298016:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298020:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298025:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298030:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298034:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298039:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298043:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298049:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298053:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298059:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298064:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298069:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298074:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298080:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572564.298085:0:40043:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:1.0:1788572564.889333:0:111606:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:1.0:1788572564.889337:0:111606:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:1.0:1788572564.889339:0:111606:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:1.0:1788572564.889340:0:111606:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:1.0:1788572564.889546:0:111606:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc5106e000 00000080:01000004:1.0:1788572564.889620:0:111606:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:1.0:1788572564.889639:0:111606:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:1.0:1788572564.889681:0:111606:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:1.0:1788572564.889695:0:111606:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc49934d40) 00000020:00000080:1.0:1788572564.889709:0:111606:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:1.0:1788572564.890040:0:111606:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572564.890064:0:111606:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:1.0:1788572564.890613:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:1.0:1788572564.890622:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:1.0:1788572564.891386:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:1.0:1788572564.891396:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572564.892117:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:2.0:1788572564.892125:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572564.893063:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572564.893071:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572564.893785:0:111606:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572564.894121:0:111606:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572564.894325:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572564.894331:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:3.0:1788572564.895252:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:3.0:1788572564.895261:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:3.0:1788572564.895706:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:3.0:1788572564.895713:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:3.0:1788572564.896078:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:3.0:1788572564.896085:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:1.0:1788572564.896650:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:1.0:1788572564.896657:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:3.0:1788572564.897567:0:111606:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:3.0:1788572564.897576:0:111606:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:0.0:1788572564.900006:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572564.900012:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572564.900014:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572564.900458:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572564.900461:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572564.900463:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:2.0:1788572564.901025:0:111606:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc58332000@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572564.901255:0:111606:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:2.0:1788572564.901265:0:111606:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:3.0:1788572564.902643:0:111620:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:2.0:1788572564.908345:0:111606:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid 99fd6a18-73c1-4957-8146-951c2385ea3d) 00000020:00000080:2.0:1788572564.908834:0:111606:0:(genops.c:1385:class_connect()) connect: client 99fd6a18-73c1-4957-8146-951c2385ea3d, cookie 0x58a604dcef352ad0 00000100:00080000:2.0:1788572564.908853:0:111606:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:2.0:1788572564.908866:0:111606:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc58332000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572564.908905:0:111606:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449863929728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572564.908936:0:111606:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 99fd6a18-73c1-4957-8146-951c2385ea3d->MGS 00000080:00200000:2.0:1788572564.908956:0:111606:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc5106e000 (sb ffff92fc5106e000) 00000080:01000000:2.0:1788572564.909041:0:111606:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:2.0:1788572564.909044:0:111606:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:2.0:1788572564.909047:0:111606:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:2.0:1788572564.909054:0:111606:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000100:00100000:3.0:1788572564.910303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99fd6a18-73c1-4957-8146-951c2385ea3d:2392:1875449863929728:192.168.202.132@tcp:250:mount.lustre.0 10000000:01000000:3.0:1788572564.913916:0:111606:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:3.0:1788572564.913921:0:111606:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc5106e000 10000000:01000000:3.0:1788572564.913926:0:111606:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc49934d40 10000000:01000000:3.0:1788572564.913930:0:111606:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:3.0:1788572564.913938:0:111606:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc49934d40 from 1 10000000:01000000:3.0:1788572564.913940:0:111606:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788572564.913985:0:111606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.913998:0:111606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.914010:0:111606:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788572564.914039:0:111606:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6733b100 x1875449863929856/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572564.914050:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 0 seconds 00010000:00010000:0.0:1788572564.918283:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572564.922565:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572564.922583:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:3.0:1788572564.922592:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:3.0:1788572564.922597:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:3.0:1788572564.922602:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58332000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:3.0:1788572564.922605:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572564.922622:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:3.0:1788572564.922634:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000003efc23e6): req@ffff92fc6733b100 x1875449863929856/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572564.922653:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99fd6a18-73c1-4957-8146-951c2385ea3d:2392:1875449863929728:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:3.0:1788572564.922800:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572564.932704:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863929856:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572564.932755:0:111606:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.932767:0:111606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572564.932844:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863929984:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572564.932933:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572564.935259:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863929984:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:3.0:1788572564.935302:0:111606:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:3.0:1788572564.935327:0:111606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.935338:0:111606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572564.935347:0:111606:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc5106e000 10000000:01000000:3.0:1788572564.935353:0:111606:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:3.0:1788572564.935358:0:111606:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc5106e000 10000000:01000000:3.0:1788572564.935360:0:111606:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:3.0:1788572564.935364:0:111606:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc5106e000 10000000:01000000:3.0:1788572564.935366:0:111606:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:3.0:1788572564.935370:0:111606:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc5106e000 from 1 10000000:01000000:3.0:1788572564.935373:0:111606:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:3.0:1788572564.935418:0:111606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.935427:0:111606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.935437:0:111606:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572564.935468:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930112:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572564.935533:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572564.937761:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930112:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572564.937790:0:111606:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572564.937808:0:111606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572564.937878:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930240:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572564.937988:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572564.955813:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930240:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572564.955892:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930368:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572564.956006:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572564.970916:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930368:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572564.972309:0:111625:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111625:1875449863930496:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572564.972597:0:111625:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc433fe600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572564.981637:0:111625:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111625:1875449863930496:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572564.981699:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572564.981711:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.981714:0:111625:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:0.0:1788572564.981742:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc5106e000 00000020:00000080:0.0:1788572564.981746:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572564.981760:0:111625:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc5106e000 (ffff92fc51cdcd40) 00000020:00000080:0.0:1788572564.981781:0:111625:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:0.0:1788572564.981787:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc5106e000 00000020:00000080:0.0:1788572564.981789:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572564.982035:0:111625:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:0.0:1788572564.982040:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572564.982044:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.982060:0:111625:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:0.0:1788572564.982064:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572564.982067:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.982069:0:111625:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572564.982072:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc5106e000 00000020:00000080:0.0:1788572564.982074:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572564.982081:0:111625:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc5106e000 (ffff92fc51cd8000) 00000020:00000080:0.0:1788572564.982092:0:111625:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:0.0:1788572564.982096:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc5106e000 00000020:00000080:0.0:1788572564.982098:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572564.986361:0:111625:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:0.0:1788572564.986389:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572564.986397:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.986398:0:111625:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572564.986402:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572564.987762:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.987767:0:111625:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:0.0:1788572564.987775:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:0.0:1788572564.987777:0:111625:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:0.0:1788572564.987779:0:111625:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:0.0:1788572564.987786:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572564.987788:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.987790:0:111625:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:0.0:1788572564.987792:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572564.987795:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572564.987796:0:111625:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:0.0:1788572564.987799:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572564.987804:0:111625:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572564.987831:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc5106e000 00000020:00000080:0.0:1788572564.987833:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572564.987847:0:111625:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc5106e000 (ffff92fc51cd99c0) 00000020:00000080:0.0:1788572564.987866:0:111625:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:0.0:1788572564.987870:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc5106e000 00000020:00000080:0.0:1788572564.987873:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572564.987951:0:111625:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc51d2d000@lustre-MDT0000-mdc-ffff92fc5106e000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572564.999150:0:111625:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc5106e000-1 is initialized 00000040:01000000:0.0:1788572565.002062:0:111625:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc5106e000 ctxt 13 is initialized 00000020:00000080:0.0:1788572565.004347:0:111625:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:0.0:1788572565.004367:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc5106e000 00000020:00000080:0.0:1788572565.004372:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572565.005239:0:111625:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:0.0:1788572565.005302:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.005309:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.005311:0:111625:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:0.0:1788572565.005314:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572565.005319:0:111625:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:0.0:1788572565.005320:0:111625:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572565.005322:0:111625:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572565.005323:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572565.005325:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.005327:0:111625:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572565.005329:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572565.005333:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.005335:0:111625:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:0.0:1788572565.005338:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572565.005345:0:111625:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572565.005376:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc5106e000 00000020:00000080:0.0:1788572565.005379:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572565.005390:0:111625:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc5106e000 (ffff92fc51cdb380) 00000020:00000080:0.0:1788572565.005418:0:111625:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:0.0:1788572565.005425:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc5106e000 00000020:00000080:0.0:1788572565.005427:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572565.007819:0:111625:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc51d2c800@lustre-MDT0001-mdc-ffff92fc5106e000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572565.011025:0:111625:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc5106e000-2 is initialized 00000040:01000000:0.0:1788572565.011606:0:111625:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc5106e000 ctxt 13 is initialized 00000020:00000080:0.0:1788572565.013336:0:111625:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:0.0:1788572565.013353:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc5106e000 00000020:00000080:0.0:1788572565.013357:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572565.013385:0:111625:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:0.0:1788572565.014296:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.014304:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.014306:0:111625:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:0.0:1788572565.014309:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572565.014312:0:111625:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:0.0:1788572565.014314:0:111625:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572565.014315:0:111625:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572565.014317:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572565.014319:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.014321:0:111625:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572565.014323:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572565.014327:0:111625:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:0.0:1788572565.014330:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.014331:0:111625:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:0.0:1788572565.014333:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572565.014338:0:111625:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572565.014362:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc5106e000 00000020:00000080:0.0:1788572565.014364:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572565.014376:0:111625:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc5106e000 (ffff92fc700319c0) 00000020:00000080:0.0:1788572565.014393:0:111625:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:0.0:1788572565.014398:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc5106e000 00000020:00000080:0.0:1788572565.014401:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572565.015294:0:111625:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990d800@lustre-OST0000-osc-ffff92fc5106e000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572565.017276:0:111625:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc5106e000-3 is initialized 00000020:00000080:0.0:1788572565.017901:0:111625:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:0.0:1788572565.017914:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc5106e000 00000020:00000080:0.0:1788572565.017918:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572565.017942:0:111625:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:0.0:1788572565.017949:0:111625:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc5106e000: Adding target lustre-OST0000_UUID 00020000:01000000:0.0:1788572565.017955:0:111625:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:0.0:1788572565.017966:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.017970:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.017972:0:111625:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:0.0:1788572565.017975:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572565.017991:0:111625:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:0.0:1788572565.017993:0:111625:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572565.017995:0:111625:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572565.017997:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572565.018000:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.018001:0:111625:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572565.018004:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572565.018008:0:111625:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:0.0:1788572565.018011:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.018012:0:111625:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:0.0:1788572565.018015:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572565.018020:0:111625:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572565.018046:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc5106e000 00000020:00000080:0.0:1788572565.018048:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572565.018058:0:111625:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc5106e000 (ffff92fc70030000) 00000020:00000080:0.0:1788572565.018070:0:111625:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:0.0:1788572565.018075:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc5106e000 00000020:00000080:0.0:1788572565.018077:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572565.018141:0:111625:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990f800@lustre-OST0001-osc-ffff92fc5106e000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572565.021110:0:111625:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc5106e000-4 is initialized 00000020:00000080:0.0:1788572565.025517:0:111625:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:0.0:1788572565.025534:0:111625:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc5106e000 00000020:00000080:0.0:1788572565.025538:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572565.025562:0:111625:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:0.0:1788572565.025572:0:111625:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc5106e000: Adding target lustre-OST0001_UUID 00020000:01000000:0.0:1788572565.025577:0:111625:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:0.0:1788572565.025588:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.025591:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.025593:0:111625:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:0.0:1788572565.025597:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572565.025601:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.025602:0:111625:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:0.0:1788572565.025606:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:0.0:1788572565.025608:0:111625:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:0.0:1788572565.025611:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.025613:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.025614:0:111625:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:0.0:1788572565.025673:0:111625:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111625:1875449863930624:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572565.025817:0:111625:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.044809:0:111625:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111625:1875449863930624:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572565.044879:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572565.044890:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572565.044893:0:111625:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788572565.044898:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788572565.044938:0:111625:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111625:1875449863930752:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572565.045146:0:111625:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49f81840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.048908:0:111625:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111625:1875449863930752:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572565.048953:0:111625:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572565.048962:0:111625:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572565.048964:0:111625:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:3.0:1788572565.048969:0:111625:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:1.0:1788572565.049133:0:111606:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:1.0:1788572565.049152:0:111606:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1788572565.049186:0:111606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572565.049197:0:111606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572565.049205:0:111606:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc5106e000 from 1 10000000:01000000:1.0:1788572565.049208:0:111606:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788572565.049246:0:111606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572565.049254:0:111606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572565.049267:0:111606:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572565.049289:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930880:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788572565.049357:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49f81a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.053553:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863930880:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572565.053587:0:111606:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572565.053601:0:111606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572565.056764:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931008:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788572565.057373:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.062895:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931008:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:3.0:1788572565.064009:0:111606:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:3.0:1788572565.064078:0:111606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572565.064092:0:111606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572565.064101:0:111606:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc5106e000 from 1 10000000:01000000:3.0:1788572565.064104:0:111606:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788572565.064145:0:111606:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572565.064153:0:111606:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572565.064164:0:111606:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572565.064188:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931136:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572565.064279:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.068966:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931136:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572565.068999:0:111606:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572565.069011:0:111606:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572565.069092:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931264:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572565.069262:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.072682:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931264:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572565.072875:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931392:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572565.073010:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc42bd8780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.076326:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:99fd6a18-73c1-4957-8146-951c2385ea3d:111606:1875449863931392:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788572565.076992:0:111630:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111630:1875449863931520:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572565.077127:0:111630:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572565.080812:0:111630:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111630:1875449863931520:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572565.080863:0:111630:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572565.080876:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.080878:0:111630:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:0.0:1788572565.080886:0:111630:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc5106e000 00000020:00000080:0.0:1788572565.080889:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:0.0:1788572565.080897:0:111630:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.080900:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.080901:0:111630:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:0.0:1788572565.080903:0:111630:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572565.080906:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.080907:0:111630:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:0.0:1788572565.080910:0:111630:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc5106e000 00000020:00000080:0.0:1788572565.080912:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:0.0:1788572565.081021:0:111630:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.081025:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.081026:0:111630:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:0.0:1788572565.081028:0:111630:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572565.081030:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.081031:0:111630:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:0.0:1788572565.081034:0:111630:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc5106e000 00000020:00000080:0.0:1788572565.081036:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:0.0:1788572565.081551:0:111630:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 508us 00000100:00100000:0.0:1788572565.081597:0:111630:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111630:1875449863931648:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572565.081701:0:111630:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572565.084494:0:111630:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:99fd6a18-73c1-4957-8146-951c2385ea3d:111630:1875449863931648:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:0.0:1788572565.084537:0:111630:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572565.084546:0:111630:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572565.084548:0:111630:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:0.0:1788572565.084553:0:111630:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:3.0:1788572565.084836:0:111606:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:3.0:1788572565.084855:0:111606:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788572565.084872:0:111606:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572565.084884:0:111606:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:3.0:1788572565.084904:0:111606:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:3.0:1788572565.084927:0:111606:0:(genops.c:1385:class_connect()) connect: client 22cad3a0-8b0a-4d3f-946a-a00629c7b71a, cookie 0x58a604dcef352b1d 00800000:01000000:3.0:1788572565.084952:0:111606:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 22cad3a0-8b0a-4d3f-946a-a00629c7b71a to lustre-clilmv-ffff92fc5106e000 00800000:01000000:3.0:1788572565.084959:0:111606:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc5106e000(22cad3a0-8b0a-4d3f-946a-a00629c7b71a) - lustre-MDT0000_UUID, 22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:3.0:1788572565.084967:0:111606:0:(genops.c:1385:class_connect()) connect: client 22cad3a0-8b0a-4d3f-946a-a00629c7b71a, cookie 0x58a604dcef352b24 00000100:00080000:3.0:1788572565.084975:0:111606:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc5106e000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572565.084983:0:111606:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc5106e000: import ffff92fc51d2d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572565.084996:0:111606:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572565.085022:0:111606:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863931776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572565.085043:0:111606:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-MDT0000_UUID 00800000:01000000:3.0:1788572565.085134:0:111606:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc5106e000(22cad3a0-8b0a-4d3f-946a-a00629c7b71a) successfully (5) 00800000:00000004:3.0:1788572565.085138:0:111606:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc5106e000: age 3320, max_age 0 00800000:01000000:3.0:1788572565.085175:0:111606:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc5106e000(22cad3a0-8b0a-4d3f-946a-a00629c7b71a) - lustre-MDT0001_UUID, 22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:3.0:1788572565.085182:0:111606:0:(genops.c:1385:class_connect()) connect: client 22cad3a0-8b0a-4d3f-946a-a00629c7b71a, cookie 0x58a604dcef352b2b 00000100:00080000:3.0:1788572565.085188:0:111606:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc5106e000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572565.085193:0:111606:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc5106e000: import ffff92fc51d2c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572565.085199:0:111606:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572565.085210:0:111606:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863932032/t0(0) o38->lustre-MDT0001-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572565.085218:0:111606:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-MDT0001_UUID 00800000:01000000:3.0:1788572565.085257:0:111606:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc5106e000(22cad3a0-8b0a-4d3f-946a-a00629c7b71a) successfully (5) 00800000:00000004:3.0:1788572565.085260:0:111606:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc5106e000: age 3320, max_age 0 00000080:00000004:3.0:1788572565.085276:0:111606:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc5106e000: age 3320, max_age 4319 00800000:00000004:3.0:1788572565.085281:0:111606:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc5106e000: age 3320, max_age 4319 00000100:00080000:3.0:1788572565.085298:0:111606:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67338380 x1875449863932288/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572565.085308:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4816f600 going to sleep for 0 seconds 00000100:00080000:3.0:1788572565.085356:0:2396:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6733a300 x1875449863931904/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572565.085368:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1788572565.085373:0:2396:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67339880 x1875449863932160/t0(0) o41->lustre-MDT0001-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572565.085568:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863931776:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572565.085668:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863932032:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572565.089083:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5106e000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572565.089101:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc5106e000: connect to target with instance 1 00000100:00080000:3.0:1788572565.089110:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc5106e000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572565.089123:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:3.0:1788572565.089127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:3.0:1788572565.092952:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000d3a9c12b): req@ffff92fc67338380 x1875449863932288/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572565.092984:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000001f6065c3): req@ffff92fc6733a300 x1875449863931904/t0(0) o41->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572565.092999:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863931776:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572565.093038:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5106e000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572565.093050:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc5106e000: connect to target with instance 1 00000100:00080000:3.0:1788572565.093060:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc5106e000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572565.093072:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:3.0:1788572565.093076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2c800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:2.0:1788572565.093211:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4816f600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.103877:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863932032:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572565.108598:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932288:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:2.0:1788572565.108630:0:111606:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc5106e000: update ffff92fc51cd9e18 cache blocks 317011/353529 objects 1023703/1024000 00000080:00000004:2.0:1788572565.108639:0:111606:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc5106e000: update ffff92fc51cd8458 cache blocks 317011/353529 objects 1023703/1024000 00000100:00100000:2.0:1788572565.108704:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932416:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:2.0:1788572565.108777:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51d450c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.111359:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:0:1875449863932160:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572565.113206:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932416:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:3.0:1788572565.113233:0:111606:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:3.0:1788572565.113246:0:111606:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:3.0:1788572565.113251:0:111606:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:3.0:1788572565.113263:0:111606:0:(genops.c:1385:class_connect()) connect: client 22cad3a0-8b0a-4d3f-946a-a00629c7b71a, cookie 0x58a604dcef352b32 00000020:00000080:3.0:1788572565.113282:0:111606:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352b39 00000100:00080000:3.0:1788572565.113290:0:111606:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc5106e000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572565.113298:0:111606:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc5106e000: import ffff92fc4990d800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572565.113310:0:111606:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572565.113336:0:111606:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863932544/t0(0) o8->lustre-OST0000-osc-ffff92fc5106e000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572565.113369:0:111606:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-OST0000_UUID 00020000:01000000:3.0:1788572565.113386:0:111606:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc5106e000) active 00000020:00000080:3.0:1788572565.113413:0:111606:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352b40 00000100:00080000:3.0:1788572565.113418:0:111606:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc5106e000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572565.113423:0:111606:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc5106e000: import ffff92fc4990f800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572565.113428:0:111606:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572565.113570:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863932544:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572565.114019:0:111606:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863932672/t0(0) o8->lustre-OST0001-osc-ffff92fc5106e000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572565.114040:0:111606:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-OST0001_UUID 00020000:01000000:3.0:1788572565.114049:0:111606:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc5106e000) active 00000100:00100000:3.0:1788572565.114095:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932800:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788572565.114127:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4816f600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572565.114239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:0:1875449863931904:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572565.114309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863932672:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572565.119843:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc5106e000: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572565.119859:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc5106e000: connect to target with instance 1 00000100:00080000:0.0:1788572565.119870:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc5106e000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788572565.119882:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:0.0:1788572565.119886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990d800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:0.0:1788572565.119893:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:0.0:1788572565.119898:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788572565.119903:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:0.0:1788572565.119918:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863932544:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572565.124350:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc5106e000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572565.124369:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc5106e000: connect to target with instance 1 00000100:00080000:1.0:1788572565.124379:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc5106e000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572565.124391:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572565.124397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990f800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788572565.124415:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:1.0:1788572565.124420:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572565.124424:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572565.124595:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932800:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788572565.124686:0:111606:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932928:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:3.0:1788572565.124829:0:111606:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc4816f600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572565.124889:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863932672:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572565.129759:0:111606:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111606:1875449863932928:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:3.0:1788572565.129953:0:111606:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d8908) 00000080:00200000:3.0:1788572565.129962:0:111606:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:3.0:1788572565.130030:0:111606:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:3.0:1788572565.130034:0:111606:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00010000:00010000:3.0:1788572565.469995:0:111661:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:3.0:1788572565.470043:0:111661:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc5106e000 00010000:00010000:3.0:1788572565.470073:0:111661:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc5106e000 00010000:00010000:3.0:1788572565.470103:0:111661:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc5106e000 00010000:00010000:3.0:1788572565.470130:0:111661:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc5106e000 00010000:00010000:0.0:1788572565.957134:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572566.984029:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572568.011370:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572569.029256:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572569.197037:0:111809:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-sec test 60: Subdirmount of encrypted dir ====== 21:42:46 (1788572566) 00010000:00010000:2.0:1788572570.064114:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572570.117260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:99fd6a18-73c1-4957-8146-951c2385ea3d:2396:1875449863933056:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.117371:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933312:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.117400:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933568:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.117444:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572570.117451:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933184:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.117564:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933440:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.122796:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933440:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.122814:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:99fd6a18-73c1-4957-8146-951c2385ea3d:2396:1875449863933056:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.122827:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933312:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.122837:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933568:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572570.126173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2396:1875449863933184:192.168.202.132@tcp:400:kworker.0 00000080:00200008:1.0:1788572570.389358:0:111844:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572570.389374:0:111844:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d8908),name=/ 00000002:00010000:1.0:1788572570.389412:0:111844:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572570.389784:0:111844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572570.389796:0:111844:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111844 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572570.389807:0:111844:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111844 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572570.389836:0:111844:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job lsof:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111844:1875449863933696:192.168.202.132@tcp:101:lsof.0 00000100:00100000:1.0:1788572570.389933:0:111844:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49f819c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572570.396239:0:111844:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job lsof:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111844:1875449863933696:192.168.202.132@tcp:101:lsof.0 00010000:00010000:3.0:1788572570.396262:0:111844:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396272:0:111844:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396284:0:111844:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396300:0:111844:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396312:0:111844:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396320:0:111844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396327:0:111844:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572570.396339:0:111844:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728ad80 x1875449863933696/t0(0) o101->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572586 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572570.396353:0:111844:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396365:0:111844:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc67272c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572570.396371:0:111844:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572570.396396:0:111844:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572570.396400:0:111844:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d8908) for lock 0x58a604dcef352b47 00000080:00010000:3.0:1788572570.396403:0:111844:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d8908) 00000080:00010000:3.0:1788572570.396406:0:111844:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352b47 from it ffff9e0380d7fcb8 00010000:00010000:3.0:1788572570.396409:0:111844:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572570.396418:0:111844:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572571.077120:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572572.101815:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572572.616306:0:111852:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572572.616431:0:111852:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d8908),name=/ 00000002:00010000:2.0:1788572572.616459:0:111852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572572.616485:0:111852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572572.616501:0:111852:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572572.616509:0:111852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572572.616518:0:111852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d8908) 00000080:00010000:2.0:1788572572.616522:0:111852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d8908) for lock 0x58a604dcef352b47 00000080:00010000:2.0:1788572572.616530:0:111852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352b47 from it ffff9e0381723cb8 00010000:00010000:2.0:1788572572.616534:0:111852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572572.616544:0:111852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200008:2.0:1788572572.616746:0:111852:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665d8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572572.616752:0:111852:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d8908),name=/ 00000002:00010000:2.0:1788572572.616760:0:111852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572572.616768:0:111852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572572.616777:0:111852:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572572.616783:0:111852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572572.616788:0:111852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d8908) 00000080:00010000:2.0:1788572572.616790:0:111852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665d8908) for lock 0x58a604dcef352b47 00000080:00010000:2.0:1788572572.616794:0:111852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352b47 from it ffff9e0381723cb8 00010000:00010000:2.0:1788572572.616797:0:111852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572572.616803:0:111852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572572.651752:0:111852:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665d8908) 00000080:00200000:1.0:1788572572.651795:0:111852:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc5106e000 (sb ffff92fc5106e000) 00010000:00010000:1.0:1788572572.651841:0:111852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651852:0:111852:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651864:0:111852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651870:0:111852:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651890:0:111852:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572572.651896:0:111852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651903:0:111852:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651909:0:111852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651915:0:111852:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651930:0:111852:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1788572572.651936:0:111852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651942:0:111852:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651948:0:111852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651957:0:111852:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572572.651964:0:111852:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:1.0:1788572572.651966:0:111852:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:3.0:1788572572.652577:0:111618:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1788572572.652587:0:111618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652595:0:111618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205400 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:3.0:1788572572.652598:0:111618:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652608:0:111618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572572.652616:0:111618:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572572.652622:0:111618:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:3.0:1788572572.652624:0:111618:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:3.0:1788572572.652627:0:111618:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:3.0:1788572572.652666:0:111618:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652673:0:111618:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572572.652697:0:111618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652704:0:111618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70205400/0x58a604dcef352ade lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793045 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652718:0:111618:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788572572.652737:0:111618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652743:0:111618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f5200 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:3.0:1788572572.652746:0:111618:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652753:0:111618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572572.652759:0:111618:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572572.652764:0:111618:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:3.0:1788572572.652766:0:111618:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:3.0:1788572572.652767:0:111618:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:3.0:1788572572.652781:0:111618:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652788:0:111618:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572572.652794:0:111618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652800:0:111618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f5200/0x58a604dcef352b0f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179304c expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652811:0:111618:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572572.652814:0:111618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652821:0:111618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70205000 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:3.0:1788572572.652824:0:111618:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652830:0:111618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572572.652836:0:111618:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572572.652841:0:111618:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:3.0:1788572572.652844:0:111618:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:3.0:1788572572.652845:0:111618:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:3.0:1788572572.652856:0:111618:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652862:0:111618:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572572.652866:0:111618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652872:0:111618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70205000/0x58a604dcef352b16 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793053 expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572572.652882:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572572.652884:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572572.652885:0:111618:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572572.652906:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:99fd6a18-73c1-4957-8146-951c2385ea3d:2395:1875449863933824:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572572.653005:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:99fd6a18-73c1-4957-8146-951c2385ea3d:2395:1875449863934080:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572572.653037:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572572.653043:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:99fd6a18-73c1-4957-8146-951c2385ea3d:2395:1875449863933952:192.168.202.132@tcp:103:ldlm_bl.0 00020000:01000000:1.0:1788572572.653262:0:111852:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc58389000 active: 1 00020000:01000000:1.0:1788572572.653265:0:111852:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc5838c000 active: 1 00020000:01000000:1.0:1788572572.653267:0:111852:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:1.0:1788572572.653272:0:111852:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc5106e000 00020000:01000000:1.0:1788572572.653273:0:111852:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc5106e000: disconnecting target lustre-OST0001-osc-ffff92fc5106e000 00000100:00080000:1.0:1788572572.653285:0:111852:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-OST0001_UUID 00010000:00010000:0.0:1788572572.653464:0:111617:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572572.653589:0:111852:0:(import.c:65:import_set_state_nolock()) ffff92fc4990f800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572572.653638:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934208:192.168.202.132@tcp:9:umount.0 00000100:00100000:2.0:1788572572.666528:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:99fd6a18-73c1-4957-8146-951c2385ea3d:2395:1875449863933952:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:3.0:1788572572.666533:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990f800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572572.666546:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:99fd6a18-73c1-4957-8146-951c2385ea3d:2395:1875449863933824:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572572.666554:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:99fd6a18-73c1-4957-8146-951c2385ea3d:2395:1875449863934080:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572572.666609:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934208:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572572.666709:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572572.666715:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc5106e000: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572572.666774:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572572.666886:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b40 00000020:00000080:1.0:1788572572.666902:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc5838c000/LOV_OSC_UUID 00000020:00000080:0.0:1788572572.666940:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838c000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc5106e000 00000020:01000000:1.0:1788572572.666949:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc5106e000 (flags='') 00000020:00000080:1.0:1788572572.666955:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572572.666964:0:111852:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc5106e000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572572.666968:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc70030000) has no exports 00000008:01000000:1.0:1788572572.667268:0:111852:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc5106e000: client import never connected 00000100:00080000:1.0:1788572572.667271:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572572.667272:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc5106e000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572572.667275:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572572.667553:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990f800 for lustre-OST0001-osc-ffff92fc5106e000 00000020:00000080:1.0:1788572572.667559:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572572.667565:0:111852:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:00000080:1.0:1788572572.667569:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc4990a000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:1.0:1788572572.667573:0:111852:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990a000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-OST0001-osc-ffff92fc5106e000 00000020:01000000:1.0:1788572572.667578:0:111852:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc5106e000 (22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00020000:01000000:1.0:1788572572.667770:0:111852:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc5106e000 00020000:01000000:1.0:1788572572.667773:0:111852:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc5106e000: disconnecting target lustre-OST0000-osc-ffff92fc5106e000 00000100:00080000:1.0:1788572572.667785:0:111852:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-OST0000_UUID 00000100:00080000:1.0:1788572572.667881:0:111852:0:(import.c:65:import_set_state_nolock()) ffff92fc4990d800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572572.667943:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934336:192.168.202.132@tcp:9:umount.0 00000100:00080000:3.0:1788572572.671028:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990d800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572572.671053:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934336:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572572.671072:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572572.671076:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc5106e000: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572572.671081:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572572.671225:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b39 00000020:00000080:1.0:1788572572.671244:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc58389000/LOV_OSC_UUID 00000020:01000000:1.0:1788572572.671354:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc5106e000 (flags='') 00000020:00000080:1.0:1788572572.671361:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572572.671385:0:111852:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc5106e000: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572572.671391:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc700319c0) has no exports 00000020:00000080:0.0:1788572572.672856:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58389000/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc5106e000 00000008:01000000:2.0:1788572572.673030:0:111852:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc5106e000: client import never connected 00000100:00080000:2.0:1788572572.673035:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:2.0:1788572572.673039:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc5106e000: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572572.673046:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572572.673383:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572572.673393:0:111852:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:00000080:2.0:1788572572.673398:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b32 00000020:00000080:2.0:1788572572.673405:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc5838e800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:1.0:1788572572.673433:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990d800 for lustre-OST0000-osc-ffff92fc5106e000 00000020:00000080:1.0:1788572572.673442:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc4990b000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:1.0:1788572572.673444:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990b000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-OST0000-osc-ffff92fc5106e000 00000020:01000000:1.0:1788572572.673449:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc5106e000 (22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000100:00080000:2.0:1788572572.673951:0:111852:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-MDT0000_UUID 00010000:00010000:2.0:1788572572.674070:0:111852:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572572.674116:0:111852:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572572.674124:0:111852:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572572.674138:0:111852:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc58c91c00 going to sleep for 0 seconds 00000100:00100000:2.0:1788572572.674149:0:111852:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job umount:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:111852:1875449863934464:192.168.202.132@tcp:103:umount.0 00010000:00010000:2.0:1788572572.674158:0:111852:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572572.674165:0:111852:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572572.674168:0:111852:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc5106e000 lock: ffff92fc702f4800/0x58a604dcef352b47 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179307d expref: -99 pid: 111844 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00080000:2.0:1788572572.675284:0:111852:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:1.0:1788572572.676589:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838e800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-clilov-ffff92fc5106e000 00000100:00100000:3.0:1788572572.677011:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934592:192.168.202.132@tcp:39:umount.0 00000100:00080000:3.0:1788572572.681344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2d000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572572.681393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934592:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572572.681471:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572572.681477:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5106e000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572572.681487:0:111852:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617ca80 x1875449863934592/t4294983777(4294983777) o39->lustre-MDT0000-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572582 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572572.681510:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572572.685459:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b24 00000020:00000080:2.0:1788572572.685478:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc5838b000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000100:00080000:2.0:1788572572.685594:0:111852:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 22cad3a0-8b0a-4d3f-946a-a00629c7b71a->lustre-MDT0001_UUID 00000020:00000080:1.0:1788572572.686254:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838b000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-MDT0000-mdc-ffff92fc5106e000 00000100:00080000:2.0:1788572572.687140:0:111852:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2c800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572572.687237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934720:192.168.202.132@tcp:39:umount.0 00000100:00080000:3.0:1788572572.690379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51d2c800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572572.690457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:22cad3a0-8b0a-4d3f-946a-a00629c7b71a:2392:1875449863934720:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572572.690567:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:2.0:1788572572.690572:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5106e000: committing for last_committed 0 gen 2 00000100:00100000:2.0:1788572572.690581:0:111852:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc4617ca80 x1875449863934720/t4294976770(4294976770) o39->lustre-MDT0001-mdc-ffff92fc5106e000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572582 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572572.690606:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572572.693581:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b2b 00000020:00000080:2.0:1788572572.693600:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc58389800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:2.0:1788572572.693691:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b1d 00000020:00000080:2.0:1788572572.693696:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc5838b800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:1.0:1788572572.693806:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58389800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-MDT0001-mdc-ffff92fc5106e000 00000020:01000000:2.0:1788572572.693979:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc5106e000 (flags='') 00000020:00000080:2.0:1788572572.694006:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572572.694012:0:111852:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc5106e000: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572572.694017:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc51cdcd40) has no exports 00000020:00000080:2.0:1788572572.694131:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572572.694138:0:111852:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:00000080:2.0:1788572572.694142:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc51d2c000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:2.0:1788572572.694144:0:111852:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2c000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-clilov-ffff92fc5106e000 00000020:01000000:2.0:1788572572.694150:0:111852:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc5106e000 (22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:2.0:1788572572.694473:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc5106e000 (flags='') 00000020:00000080:2.0:1788572572.694477:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572572.694481:0:111852:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc5106e000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572572.694484:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc51cd8000) has no exports 00000020:00000080:2.0:1788572572.694525:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572572.694529:0:111852:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:2.0:1788572572.694533:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc5106e000 (flags='') 00000020:00000080:2.0:1788572572.694536:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572572.694539:0:111852:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc5106e000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572572.694541:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc51cd99c0) has no exports 00000020:00000080:1.0:1788572572.694980:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838b800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-clilmv-ffff92fc5106e000 00000020:00000080:1.0:1788572572.694986:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc51d2a800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:1.0:1788572572.694989:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2a800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-clilmv-ffff92fc5106e000 00000020:01000000:1.0:1788572572.694993:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc5106e000 (22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000008:01000000:2.0:1788572572.695300:0:111852:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc5106e000: client import never connected 00000100:00080000:2.0:1788572572.695306:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572572.695310:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc5106e000: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572572.695398:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572572.700258:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572572.700271:0:111852:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000020:01000000:2.0:1788572572.700277:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc5106e000 (flags='') 00000020:00000080:2.0:1788572572.700282:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572572.700285:0:111852:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc5106e000: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572572.700291:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc51cdb380) has no exports 00000020:00000080:1.0:1788572572.700414:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51d2d000 for lustre-MDT0000-mdc-ffff92fc5106e000 00000020:00000080:1.0:1788572572.700424:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc51d2b000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:1.0:1788572572.700427:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2b000/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-MDT0000-mdc-ffff92fc5106e000 00000020:01000000:1.0:1788572572.700435:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc5106e000 (22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000008:01000000:2.0:1788572572.701599:0:111852:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc5106e000: client import never connected 00000100:00080000:2.0:1788572572.701606:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:2.0:1788572572.701610:0:111852:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc5106e000: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572572.701619:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572572.714888:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572572.714903:0:111852:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc5106e000 (uuid 22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000080:02000400:0.0:1788572572.714908:0:111852:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:0.0:1788572572.714911:0:111852:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:0.0:1788572572.714957:0:111852:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc5106e000 00000100:00080000:0.0:1788572572.714966:0:111852:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 99fd6a18-73c1-4957-8146-951c2385ea3d->MGS 00010000:00010000:0.0:1788572572.714984:0:111852:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572572.714997:0:111852:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572572.715005:0:111852:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572572.715008:0:111852:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:0.0:1788572572.715011:0:111852:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:0.0:1788572572.715056:0:111852:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572572.715063:0:111852:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572572.715084:0:111852:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053000 going to sleep for 0 seconds 00000100:00100000:0.0:1788572572.715097:0:111852:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job umount:99fd6a18-73c1-4957-8146-951c2385ea3d:111852:1875449863934848:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572572.715107:0:111852:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572572.715116:0:111852:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572572.715119:0:111852:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70204800/0x58a604dcef352ad7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179303e expref: -99 pid: 111606 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572572.715147:0:111852:0:(import.c:65:import_set_state_nolock()) ffff92fc58332000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1788572572.715175:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99fd6a18-73c1-4957-8146-951c2385ea3d:2392:1875449863934976:192.168.202.132@tcp:251:umount.0 00000020:00000080:0.0:1788572572.718897:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51d2c800 for lustre-MDT0001-mdc-ffff92fc5106e000 00000020:00000080:0.0:1788572572.718918:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc51d2b800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a 00000020:00000080:0.0:1788572572.718923:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51d2b800/22cad3a0-8b0a-4d3f-946a-a00629c7b71a for lustre-MDT0001-mdc-ffff92fc5106e000 00000020:01000000:0.0:1788572572.718933:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc5106e000 (22cad3a0-8b0a-4d3f-946a-a00629c7b71a) 00000100:00080000:3.0:1788572572.754093:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58332000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:3.0:1788572572.754106:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572572.754109:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572572.754111:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:3.0:1788572572.754463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:99fd6a18-73c1-4957-8146-951c2385ea3d:2392:1875449863934976:192.168.202.132@tcp:251:umount.0 00000100:00080000:2.0:1788572572.754520:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572572.754524:0:111852:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572572.754527:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572572.754529:0:111852:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572572.754539:0:111852:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352ad0 00000020:00000080:2.0:1788572572.754548:0:111852:0:(genops.c:885:class_export_put()) final put ffff92fc58335000/99fd6a18-73c1-4957-8146-951c2385ea3d 00000020:01000000:2.0:1788572572.754573:0:111852:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:2.0:1788572572.754579:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572572.754587:0:111852:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572572.754591:0:111852:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc49934d40) has no exports 00000020:00000080:3.0:1788572572.754691:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc58335000/99fd6a18-73c1-4957-8146-951c2385ea3d for MGC192.168.202.132@tcp 10000000:01000000:1.0:1788572572.754974:0:111620:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788572572.755123:0:111852:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:2.0:1788572572.755126:0:111852:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572572.755129:0:111852:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572572.755131:0:111852:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572572.755133:0:111852:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572572.755294:0:111852:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572572.755300:0:111852:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid 99fd6a18-73c1-4957-8146-951c2385ea3d) 00000020:01000004:2.0:1788572572.755304:0:111852:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc5106e000 1 00000020:01000004:2.0:1788572572.755308:0:111852:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc5106f000 00000020:00000080:3.0:1788572572.755452:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc58332000 for MGC192.168.202.132@tcp 00000020:00000080:3.0:1788572572.755460:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc51069800/99fd6a18-73c1-4957-8146-951c2385ea3d 00000020:00000080:3.0:1788572572.755462:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51069800/99fd6a18-73c1-4957-8146-951c2385ea3d for MGC192.168.202.132@tcp 00000020:01000000:3.0:1788572572.755467:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (99fd6a18-73c1-4957-8146-951c2385ea3d) 00010000:00010000:1.0:1788572572.755846:0:111618:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572572.756147:0:111617:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00000100:00100000:1.0:1788572572.756500:0:111608:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc49d31c00:111608] 1 exiting: rc = 0 00000100:00100000:0.0:1788572572.756677:0:111607:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc49d31600:111607] 0 exiting: rc = 1 00000100:00100000:3.0:1788572572.757314:0:111609:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc47f04200:111609] 0 exiting: rc = 1 00000100:00100000:3.0:1788572572.757386:0:111610:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff92fc491ec200:111610] 1 exiting: rc = 0 00000100:00100000:0.0:1788572572.757786:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.757796:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.758376:0:111612:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc470f3a00:111612] 1 exiting: rc = 0 00000100:00100000:0.0:1788572572.758437:0:111613:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc470f2200:111613] 2 exiting: rc = 0 00000100:00100000:1.0:1788572572.758630:0:111611:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc491eca00:111611] 0 exiting: rc = 1 00000100:00100000:3.0:1788572572.758834:0:111614:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc4137fe00:111614] 0 exiting: rc = 1 00000100:00100000:3.0:1788572572.758960:0:111616:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc53ea5000:111616] 2 exiting: rc = 0 00000100:00100000:2.0:1788572572.759390:0:111615:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc51e43800:111615] 1 exiting: rc = 0 00000100:00100000:0.0:1788572572.761983:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.761994:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.761999:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762005:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762010:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762016:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762020:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762026:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762030:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762035:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762040:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762045:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762050:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762056:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762061:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762066:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762072:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762078:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762083:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762088:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762094:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762099:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762104:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762109:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762115:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762121:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762127:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762132:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762138:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762144:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762149:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762154:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762159:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762164:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762169:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762174:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762180:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762261:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762268:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762273:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762279:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762284:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762290:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762294:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762299:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762304:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762309:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762314:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762320:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762325:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762330:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762336:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762341:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762347:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762353:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762358:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762373:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762378:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762383:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762388:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762394:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762400:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762406:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762411:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762419:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762425:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762431:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762436:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762441:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762446:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762452:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762459:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762464:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762469:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762474:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762479:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762484:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762490:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762496:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762501:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762506:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762511:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762516:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762521:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762526:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762530:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762535:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762540:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762545:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762550:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762556:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762561:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762567:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762573:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762579:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762584:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762590:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762595:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762600:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762606:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762612:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762617:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762623:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762628:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762633:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762638:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762643:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762649:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762654:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762659:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762665:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762671:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762676:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762681:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762686:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762692:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762697:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762702:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762707:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762712:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762717:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762742:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762748:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762754:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762761:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762768:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762774:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:0.0:1788572572.762779:0:31488:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:2.0:1788572573.001053:0:111865:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:2.0:1788572573.001058:0:111865:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:2.0:1788572573.001059:0:111865:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre 00000020:01000004:2.0:1788572573.001061:0:111865:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock,test_dummy_encryption 00000080:01200004:2.0:1788572573.001332:0:111865:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc48718800 00000080:01000004:2.0:1788572573.001341:0:111865:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:2.0:1788572573.001371:0:111865:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:2.0:1788572573.001400:0:111865:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:2.0:1788572573.001414:0:111865:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc48cc8000) 00000020:00000080:2.0:1788572573.001428:0:111865:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:2.0:1788572573.001616:0:111865:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788572573.001640:0:111865:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788572573.003235:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:2.0:1788572573.003249:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572573.005415:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572573.005428:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572573.006491:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:2.0:1788572573.006499:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:0.0:1788572573.006856:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572573.006862:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:0.0:1788572573.007497:0:111865:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572573.007799:0:111865:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:0.0:1788572573.007983:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:0.0:1788572573.007988:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:2.0:1788572573.010586:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:2.0:1788572573.010603:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:0.0:1788572573.019898:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:0.0:1788572573.019921:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:1.0:1788572573.021009:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:1.0:1788572573.021021:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:0.0:1788572573.021775:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:0.0:1788572573.021786:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:0.0:1788572573.023432:0:111865:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:0.0:1788572573.023443:0:111865:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:0.0:1788572573.024667:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572573.024671:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572573.024673:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.025588:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.025592:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.025594:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788572573.029955:0:111865:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc42c0f800@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572573.030119:0:111865:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:1.0:1788572573.030127:0:111865:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:1.0:1788572573.030828:0:111879:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:0.0:1788572573.030901:0:111865:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid c727f535-e4fd-4a83-8271-2859b65ffa9f) 00000020:00000080:0.0:1788572573.030923:0:111865:0:(genops.c:1385:class_connect()) connect: client c727f535-e4fd-4a83-8271-2859b65ffa9f, cookie 0x58a604dcef352b55 00000100:00080000:0.0:1788572573.030936:0:111865:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572573.030945:0:111865:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc42c0f800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572573.030982:0:111865:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449863935104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572573.031014:0:111865:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import c727f535-e4fd-4a83-8271-2859b65ffa9f->MGS 00000080:00200000:0.0:1788572573.031033:0:111865:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc48718800 (sb ffff92fc48718800) 00000080:01000000:0.0:1788572573.031092:0:111865:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock,test_dummy_encryption 00000080:00000004:0.0:1788572573.031095:0:111865:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788572573.031099:0:111865:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:00000004:0.0:1788572573.031101:0:111865:0:(llite_lib.c:1242:ll_options()) next opt=test_dummy_encryption 00000080:01000000:0.0:1788572573.031109:0:111865:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 8dd79b58-105c-49b4-a0c4-61b51a672039 00000100:00100000:1.0:1788572573.031402:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c727f535-e4fd-4a83-8271-2859b65ffa9f:2392:1875449863935104:192.168.202.132@tcp:250:mount.lustre.0 10000000:01000000:0.0:1788572573.034666:0:111865:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788572573.034672:0:111865:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc48718800 10000000:01000000:0.0:1788572573.034677:0:111865:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc48cc8000 10000000:01000000:0.0:1788572573.034681:0:111865:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572573.034690:0:111865:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc48cc8000 from 1 10000000:01000000:0.0:1788572573.034692:0:111865:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788572573.034768:0:111865:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.034781:0:111865:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.034794:0:111865:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572573.034825:0:111865:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704bb80 x1875449863935232/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572573.034836:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ae40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572573.035093:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572573.067251:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572573.067268:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:2.0:1788572573.067278:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:2.0:1788572573.067285:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:2.0:1788572573.067290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0f800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:2.0:1788572573.067293:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572573.067359:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:2.0:1788572573.067373:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000477e7183): req@ffff92fc6704bb80 x1875449863935232/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572573.067392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c727f535-e4fd-4a83-8271-2859b65ffa9f:2392:1875449863935104:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788572573.068592:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ae40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.086616:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935232:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572573.086654:0:111865:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.086669:0:111865:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.087750:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935360:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572573.087831:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45add900 going to sleep for 11 seconds 00000100:00100000:0.0:1788572573.100084:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935360:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:0.0:1788572573.100446:0:111865:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1788572573.100466:0:111865:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.100477:0:111865:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572573.100487:0:111865:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc48718800 10000000:01000000:0.0:1788572573.100496:0:111865:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:0.0:1788572573.100500:0:111865:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc48718800 10000000:01000000:0.0:1788572573.100503:0:111865:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788572573.100507:0:111865:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc48718800 10000000:01000000:0.0:1788572573.100511:0:111865:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:0.0:1788572573.100516:0:111865:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc48718800 from 1 10000000:01000000:0.0:1788572573.100519:0:111865:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788572573.100558:0:111865:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.100567:0:111865:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.100578:0:111865:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572573.100602:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935488:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572573.100699:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4a300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.110566:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935488:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572573.110600:0:111865:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572573.110615:0:111865:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572573.110701:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935616:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572573.111226:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.121784:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935616:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572573.121851:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935744:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572573.121953:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.124293:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863935744:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572573.124928:0:111884:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111884:1875449863935872:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572573.125013:0:111884:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccf000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.133874:0:111884:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111884:1875449863935872:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572573.133932:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.133944:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.133947:0:111884:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:2.0:1788572573.133957:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc48718800 00000020:00000080:2.0:1788572573.133961:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572573.133976:0:111884:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc48718800 (ffff92fc48cc99c0) 00000020:00000080:2.0:1788572573.134003:0:111884:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:2.0:1788572573.134010:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc48718800 00000020:00000080:2.0:1788572573.134013:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572573.134435:0:111884:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:2.0:1788572573.134441:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572573.134445:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.134447:0:111884:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:2.0:1788572573.134450:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.134453:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.134454:0:111884:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572573.134457:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc48718800 00000020:00000080:2.0:1788572573.134460:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572573.134467:0:111884:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc48718800 (ffff92fc48ccb380) 00000020:00000080:2.0:1788572573.134476:0:111884:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:2.0:1788572573.134480:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc48718800 00000020:00000080:2.0:1788572573.134482:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:2.0:1788572573.140096:0:111884:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:2.0:1788572573.140111:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572573.140119:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.140121:0:111884:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:2.0:1788572573.140125:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.140129:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.140131:0:111884:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:2.0:1788572573.140134:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:2.0:1788572573.140137:0:111884:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:2.0:1788572573.140139:0:111884:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:2.0:1788572573.140145:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572573.140148:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.140149:0:111884:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:2.0:1788572573.140152:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.140156:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.140157:0:111884:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:2.0:1788572573.140160:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572573.140165:0:111884:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572573.141948:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc48718800 00000020:00000080:2.0:1788572573.141953:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572573.141968:0:111884:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc48718800 (ffff92fc48cccd40) 00000020:00000080:2.0:1788572573.141987:0:111884:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:2.0:1788572573.141993:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc48718800 00000020:00000080:2.0:1788572573.141996:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572573.142083:0:111884:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc51bec800@lustre-MDT0000-mdc-ffff92fc48718800: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572573.149025:0:111884:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc48718800-1 is initialized 00000040:01000000:2.0:1788572573.152099:0:111884:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc48718800 ctxt 13 is initialized 00000020:00000080:0.0:1788572573.161038:0:111884:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:0.0:1788572573.161063:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc48718800 00000020:00000080:0.0:1788572573.161069:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572573.161096:0:111884:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:0.0:1788572573.161139:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572573.161143:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.161145:0:111884:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:0.0:1788572573.161147:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572573.161150:0:111884:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:0.0:1788572573.161152:0:111884:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572573.161153:0:111884:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572573.161154:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572573.161157:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.161159:0:111884:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572573.161161:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572573.161166:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.161959:0:111884:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:0.0:1788572573.161969:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572573.161975:0:111884:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572573.162004:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc48718800 00000020:00000080:0.0:1788572573.162008:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572573.162018:0:111884:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc48718800 (ffff92fc70033380) 00000020:00000080:0.0:1788572573.162034:0:111884:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:0.0:1788572573.162040:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc48718800 00000020:00000080:0.0:1788572573.162042:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572573.162118:0:111884:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc48dc1000@lustre-MDT0001-mdc-ffff92fc48718800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572573.167203:0:111884:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc48718800-2 is initialized 00000040:01000000:0.0:1788572573.173532:0:111884:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc48718800 ctxt 13 is initialized 00000020:00000080:0.0:1788572573.174932:0:111884:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:0.0:1788572573.174950:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc48718800 00000020:00000080:0.0:1788572573.174954:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572573.174980:0:111884:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:0.0:1788572573.175017:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572573.175021:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.175023:0:111884:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:0.0:1788572573.175026:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572573.175029:0:111884:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:0.0:1788572573.175031:0:111884:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572573.175032:0:111884:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572573.175033:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572573.175036:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.175037:0:111884:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572573.175040:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572573.175044:0:111884:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:0.0:1788572573.175047:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.175049:0:111884:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:0.0:1788572573.175051:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572573.175057:0:111884:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572573.175084:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc48718800 00000020:00000080:0.0:1788572573.175086:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572573.175097:0:111884:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc48718800 (ffff92fc70034d40) 00000020:00000080:0.0:1788572573.175112:0:111884:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:0.0:1788572573.175118:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc48718800 00000020:00000080:0.0:1788572573.175121:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572573.175207:0:111884:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc48dc5000@lustre-OST0000-osc-ffff92fc48718800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572573.175443:0:111884:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc48718800-3 is initialized 00000020:00000080:0.0:1788572573.194500:0:111884:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:0.0:1788572573.194520:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc48718800 00000020:00000080:0.0:1788572573.194524:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572573.194550:0:111884:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:0.0:1788572573.194559:0:111884:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc48718800: Adding target lustre-OST0000_UUID 00020000:01000000:0.0:1788572573.194565:0:111884:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:0.0:1788572573.194578:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572573.194582:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.194583:0:111884:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:0.0:1788572573.194588:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572573.194591:0:111884:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:0.0:1788572573.194593:0:111884:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572573.194595:0:111884:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572573.194596:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572573.194599:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.194600:0:111884:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572573.194603:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572573.194607:0:111884:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre 00000020:00000080:0.0:1788572573.194611:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.194612:0:111884:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:0.0:1788572573.194616:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572573.194621:0:111884:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572573.194650:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc48718800 00000020:00000080:0.0:1788572573.194653:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572573.194663:0:111884:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc48718800 (ffff92fc700319c0) 00000020:00000080:0.0:1788572573.194678:0:111884:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:0.0:1788572573.194684:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc48718800 00000020:00000080:0.0:1788572573.194686:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572573.194786:0:111884:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc48dc6000@lustre-OST0001-osc-ffff92fc48718800: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572573.195005:0:111884:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc48718800-4 is initialized 00000020:00000080:0.0:1788572573.195246:0:111884:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:0.0:1788572573.195254:0:111884:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc48718800 00000020:00000080:0.0:1788572573.195257:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572573.195265:0:111884:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:0.0:1788572573.195270:0:111884:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc48718800: Adding target lustre-OST0001_UUID 00020000:01000000:0.0:1788572573.195273:0:111884:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:0.0:1788572573.195280:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572573.195284:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.195285:0:111884:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:0.0:1788572573.195288:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572573.195291:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.195292:0:111884:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:0.0:1788572573.195295:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:0.0:1788572573.195297:0:111884:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:0.0:1788572573.195298:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572573.195300:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572573.195302:0:111884:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:0.0:1788572573.195359:0:111884:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111884:1875449863936000:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:0.0:1788572573.195592:0:111884:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ad80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.204821:0:111884:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111884:1875449863936000:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572573.204861:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.204872:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.204875:0:111884:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572573.204882:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572573.204918:0:111884:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111884:1875449863936128:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572573.204995:0:111884:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccff00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.217896:0:111884:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111884:1875449863936128:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572573.217945:0:111884:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572573.217956:0:111884:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.217958:0:111884:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:2.0:1788572573.217963:0:111884:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:0.0:1788572573.218117:0:111865:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:0.0:1788572573.218134:0:111865:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788572573.218154:0:111865:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.218166:0:111865:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572573.218174:0:111865:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc48718800 from 1 10000000:01000000:0.0:1788572573.218177:0:111865:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788572573.218214:0:111865:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.218223:0:111865:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.218233:0:111865:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572573.218256:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936256:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788572573.218336:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ab40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.225136:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936256:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572573.225169:0:111865:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572573.225183:0:111865:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572573.226122:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936384:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:2.0:1788572573.226265:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.240048:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936384:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:2.0:1788572573.241405:0:111865:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572573.241439:0:111865:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572573.241452:0:111865:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572573.241461:0:111865:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc48718800 from 1 10000000:01000000:2.0:1788572573.241464:0:111865:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572573.241508:0:111865:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572573.241516:0:111865:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572573.241526:0:111865:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572573.241568:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936512:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572573.241865:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfd80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572573.248632:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936512:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572573.248670:0:111865:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.248684:0:111865:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.248818:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936640:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572573.248902:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ae40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.253707:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936640:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572573.253803:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936768:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572573.253901:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccff00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.260553:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c727f535-e4fd-4a83-8271-2859b65ffa9f:111865:1875449863936768:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572573.261048:0:111892:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111892:1875449863936896:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572573.261137:0:111892:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ae40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.276990:0:111892:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111892:1875449863936896:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572573.277183:0:111892:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.277198:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.277200:0:111892:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572573.277210:0:111892:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc48718800 00000020:00000080:2.0:1788572573.277213:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572573.277221:0:111892:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572573.277223:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.277225:0:111892:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572573.277227:0:111892:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.277229:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.277230:0:111892:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572573.277233:0:111892:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc48718800 00000020:00000080:2.0:1788572573.277235:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572573.277299:0:111892:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572573.277304:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.277305:0:111892:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:2.0:1788572573.277308:0:111892:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572573.277310:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572573.277312:0:111892:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:2.0:1788572573.277316:0:111892:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc48718800 00000020:00000080:2.0:1788572573.277318:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:2.0:1788572573.278138:0:111892:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 813us 00000100:00100000:2.0:1788572573.278295:0:111892:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111892:1875449863937024:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572573.278509:0:111892:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccff00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.289816:0:111892:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c727f535-e4fd-4a83-8271-2859b65ffa9f:111892:1875449863937024:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572573.289877:0:111892:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572573.289888:0:111892:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572573.289890:0:111892:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:3.0:1788572573.289894:0:111892:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:1.0:1788572573.290380:0:111865:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:1.0:1788572573.290401:0:111865:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788572573.290428:0:111865:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.290441:0:111865:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788572573.290464:0:111865:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1788572573.290550:0:111865:0:(genops.c:1385:class_connect()) connect: client 8dd79b58-105c-49b4-a0c4-61b51a672039, cookie 0x58a604dcef352ba2 00800000:01000000:1.0:1788572573.290617:0:111865:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 8dd79b58-105c-49b4-a0c4-61b51a672039 to lustre-clilmv-ffff92fc48718800 00800000:01000000:1.0:1788572573.290624:0:111865:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc48718800(8dd79b58-105c-49b4-a0c4-61b51a672039) - lustre-MDT0000_UUID, 8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:1.0:1788572573.290630:0:111865:0:(genops.c:1385:class_connect()) connect: client 8dd79b58-105c-49b4-a0c4-61b51a672039, cookie 0x58a604dcef352ba9 00000100:00080000:1.0:1788572573.290637:0:111865:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc48718800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572573.290644:0:111865:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc48718800: import ffff92fc51bec800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572573.290654:0:111865:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572573.290675:0:111865:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449863937152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572573.290751:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863937152:192.168.202.132@tcp:38:mount.lustre.0 00000100:00080000:1.0:1788572573.291573:0:111865:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-MDT0000_UUID 00800000:01000000:1.0:1788572573.291655:0:111865:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc48718800(8dd79b58-105c-49b4-a0c4-61b51a672039) successfully (5) 00800000:00000004:1.0:1788572573.291659:0:111865:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc48718800: age 3328, max_age 0 00800000:01000000:1.0:1788572573.291693:0:111865:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc48718800(8dd79b58-105c-49b4-a0c4-61b51a672039) - lustre-MDT0001_UUID, 8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:1.0:1788572573.291701:0:111865:0:(genops.c:1385:class_connect()) connect: client 8dd79b58-105c-49b4-a0c4-61b51a672039, cookie 0x58a604dcef352bb0 00000100:00080000:1.0:1788572573.291706:0:111865:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc48718800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572573.291712:0:111865:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc48718800: import ffff92fc48dc1000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572573.291734:0:111865:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572573.291744:0:111865:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449863937408/t0(0) o38->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572573.296642:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863937408:192.168.202.132@tcp:38:mount.lustre.0 00000100:00080000:0.0:1788572573.296835:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67289880 x1875449863937280/t0(0) o41->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572573.299440:0:111865:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-MDT0001_UUID 00800000:01000000:1.0:1788572573.299689:0:111865:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc48718800(8dd79b58-105c-49b4-a0c4-61b51a672039) successfully (5) 00800000:00000004:1.0:1788572573.299695:0:111865:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc48718800: age 3328, max_age 0 00000100:00080000:1.0:1788572573.299778:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67289500 x1875449863937536/t0(0) o41->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000080:00000004:1.0:1788572573.299843:0:111865:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc48718800: age 3328, max_age 4328 00800000:00000004:1.0:1788572573.299849:0:111865:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc48718800: age 3328, max_age 4328 00000100:00080000:1.0:1788572573.299868:0:111865:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67289f80 x1875449863937664/t0(0) o41->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572573.299877:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711af00 going to sleep for 0 seconds 00000100:00100000:0.0:1788572573.308742:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc48718800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572573.308763:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc48718800: connect to target with instance 1 00000100:00080000:0.0:1788572573.308774:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc48718800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:0.0:1788572573.308789:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:0.0:1788572573.308795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc1000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00080000:0.0:1788572573.309764:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002939d3c5): req@ffff92fc67289500 x1875449863937536/t0(0) o41->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572573.309792:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863937408:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788572573.309814:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc48718800: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572573.309820:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc48718800: connect to target with instance 1 00000100:00080000:0.0:1788572573.309827:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc48718800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:0.0:1788572573.309834:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:0.0:1788572573.310000:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51bec800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:2.0:1788572573.311279:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc67289880 x1875449863937280/t0(0) o41->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572573.311306:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000071d0f635): req@ffff92fc67289f80 x1875449863937664/t0(0) o41->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572573.311321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863937152:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572573.312373:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711af00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.316252:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8dd79b58-105c-49b4-a0c4-61b51a672039:0:1875449863937536:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788572573.319078:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863937664:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:0.0:1788572573.319117:0:111865:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc48718800: update ffff92fc48ccd198 cache blocks 317011/353529 objects 1023703/1024000 00000080:00000004:0.0:1788572573.319124:0:111865:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc48718800: update ffff92fc48ccb7d8 cache blocks 317011/353529 objects 1023703/1024000 00000100:00100000:0.0:1788572573.319166:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863937792:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:0.0:1788572573.319251:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ab40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572573.321970:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863937792:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:0.0:1788572573.322000:0:111865:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:0.0:1788572573.322016:0:111865:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:0.0:1788572573.322021:0:111865:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:0.0:1788572573.322035:0:111865:0:(genops.c:1385:class_connect()) connect: client 8dd79b58-105c-49b4-a0c4-61b51a672039, cookie 0x58a604dcef352bb7 00000020:00000080:0.0:1788572573.322055:0:111865:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352bbe 00000100:00080000:0.0:1788572573.322064:0:111865:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc48718800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572573.322072:0:111865:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc48718800: import ffff92fc48dc5000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572573.322098:0:111865:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572573.322123:0:111865:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449863937920/t0(0) o8->lustre-OST0000-osc-ffff92fc48718800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572573.322158:0:111865:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-OST0000_UUID 00020000:01000000:0.0:1788572573.322185:0:111865:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc48718800) active 00000020:00000080:0.0:1788572573.322216:0:111865:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352bc5 00000100:00080000:0.0:1788572573.322221:0:111865:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc48718800: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572573.322226:0:111865:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc48718800: import ffff92fc48dc6000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572573.322231:0:111865:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572573.322242:0:111865:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449863938048/t0(0) o8->lustre-OST0001-osc-ffff92fc48718800@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572573.322250:0:111865:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-OST0001_UUID 00020000:01000000:0.0:1788572573.322255:0:111865:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc48718800) active 00000080:02000400:0.0:1788572573.322266:0:111865:0:(llite_lib.c:801:client_common_fill_super()) Test dummy encryption mode enabled 00000100:00100000:0.0:1788572573.322298:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863938176:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572573.322369:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.322478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863937920:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572573.322518:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863938048:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572573.326489:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8dd79b58-105c-49b4-a0c4-61b51a672039:0:1875449863937280:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572573.327492:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc48718800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572573.327503:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc48718800: connect to target with instance 1 00000100:00080000:3.0:1788572573.327512:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc48718800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572573.327525:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:3.0:1788572573.327537:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc6000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572573.327547:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:3.0:1788572573.327552:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572573.327556:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572573.327571:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863938048:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572573.327805:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863938176:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788572573.327875:0:111865:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863938304:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788572573.327990:0:111865:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ab40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.332584:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc48718800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572573.332603:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc48718800: connect to target with instance 1 00000100:00080000:3.0:1788572573.332614:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc48718800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572573.332626:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:3.0:1788572573.332630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572573.332637:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:3.0:1788572573.332642:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572573.332646:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572573.332659:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863937920:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:0.0:1788572573.334918:0:111865:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:8dd79b58-105c-49b4-a0c4-61b51a672039:111865:1875449863938304:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788572573.335134:0:111865:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00200000:0.0:1788572573.335144:0:111865:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:02000400:0.0:1788572573.335216:0:111865:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:0.0:1788572573.335220:0:111865:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre: Mount complete 00000080:00200000:3.0:1788572573.452470:0:111910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:3.0:1788572573.452502:0:111910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572573.452565:0:111910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111910 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.452578:0:111910:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111910 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.452593:0:111910:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111910 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572573.452625:0:111910:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111910:1875449863938432:192.168.202.132@tcp:101:mkdir.0 00000100:00100000:3.0:1788572573.452982:0:111910:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.457446:0:111910:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111910:1875449863938432:192.168.202.132@tcp:101:mkdir.0 00010000:00010000:3.0:1788572573.457480:0:111910:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457494:0:111910:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457508:0:111910:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457530:0:111910:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457544:0:111910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457553:0:111910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457561:0:111910:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572573.457574:0:111910:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67338700 x1875449863938432/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572573.457590:0:111910:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457604:0:111910:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572573.457611:0:111910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572573.457642:0:111910:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572573.457648:0:111910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352bcc 00000080:00010000:3.0:1788572573.457654:0:111910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:3.0:1788572573.457658:0:111910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bcc from it ffff9e03822d3be0 00010000:00010000:3.0:1788572573.457662:0:111910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.457676:0:111910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572573.457707:0:111910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572573.457926:0:111910:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d60.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ea1c8), flags=1536 00000080:00200000:3.0:1788572573.457933:0:111910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:3.0:1788572573.457951:0:111910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572573.457961:0:111910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572573.457973:0:111910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572573.457980:0:111910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572573.457985:0:111910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:3.0:1788572573.457988:0:111910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352bcc 00000080:00010000:3.0:1788572573.457991:0:111910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bcc from it ffff9e03822d3ce0 00010000:00010000:3.0:1788572573.457995:0:111910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572573.458002:0:111910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:3.0:1788572573.458153:0:111910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788572573.458163:0:111910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:3.0:1788572573.458172:0:111910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572573.458180:0:111910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572573.458188:0:111910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:3.0:1788572573.458195:0:111910:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572573.458198:0:111910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:3.0:1788572573.458201:0:111910:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352bcc 00000080:00010000:3.0:1788572573.458205:0:111910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bcc from it ffff9e03822d3dc8 00010000:00010000:3.0:1788572573.458208:0:111910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572573.458216:0:111910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572573.458224:0:111910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788572573.458229:0:111910:0:(namei.c:2172:do_mkdir()) VFS Op:name=d60.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000100:00100000:3.0:1788572573.481681:0:111910:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111910:1875449863938560:192.168.202.132@tcp:700:mkdir.0 00000100:00100000:3.0:1788572573.481808:0:111910:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.484390:0:111910:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111910:1875449863938560:192.168.202.132@tcp:700:mkdir.0 00010000:00010000:3.0:1788572573.484433:0:111910:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572573.484490:0:111910:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572573.484503:0:111910:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572573.484505:0:111910:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bcc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217930ca expref: -99 pid: 111910 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000020:00100000:3.0:1788572573.484531:0:111910:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc48718800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572573.484540:0:111910:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67338700 x1875449863938816/t0(0) o36->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 592/688 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572573.484563:0:111910:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111910:1875449863938816:192.168.202.132@tcp:36:mkdir.0 00000100:00100000:3.0:1788572573.484641:0:111910:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.489352:0:111910:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111910:1875449863938816:192.168.202.132@tcp:36:mkdir.0 00000080:00200000:3.0:1788572573.489397:0:111910:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08) 00000080:00200000:3.0:1788572573.489405:0:111910:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572573.514308:0:111676:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:1.0:1788572573.514354:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572573.514416:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.514429:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.514440:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.514469:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863938944:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.514598:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.518215:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863938944:192.168.202.132@tcp:101:bash.0 00010000:00010000:3.0:1788572573.518243:0:111676:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518255:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518266:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518277:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518294:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518302:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518310:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572573.518322:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67289880 x1875449863938944/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572573.518347:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518360:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572573.518367:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572573.518395:0:111676:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572573.518401:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352bd3 00000080:00010000:3.0:1788572573.518406:0:111676:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:3.0:1788572573.518409:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bd3 from it ffff9e0382023bf8 00010000:00010000:3.0:1788572573.518413:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518421:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572573.518451:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572573.518467:0:111676:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d60.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ea1c8), flags=25 00000080:00200000:3.0:1788572573.518472:0:111676:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d60.sanity-sec, dir=[0x200000007:0x1:0x0](ffff92fc665ea1c8), intent=0 00000002:00010000:3.0:1788572573.518492:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d60.sanity-sec,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788572573.518530:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518537:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.518546:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572573.518566:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939072:192.168.202.132@tcp:101:bash.0 00000100:00100000:3.0:1788572573.518639:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.522241:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939072:192.168.202.132@tcp:101:bash.0 00010000:00010000:3.0:1788572573.522269:0:111676:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522290:0:111676:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522298:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522310:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522322:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522345:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522354:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522362:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572573.522373:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc6733b800 x1875449863939072/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 584/616 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572573.522386:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522396:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 8589935650/1 (0 0) 00000080:00200000:3.0:1788572573.522402:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572573.522425:0:111676:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000080:00200000:3.0:1788572573.522429:0:111676:0:(namei.c:147:ll_iget()) got inode: [0x200000422:0x1:0x0](ffff92fc665dcb08): rc = 0 00000080:00010000:3.0:1788572573.522439:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dcb08) for lock 0x58a604dcef352bda 00000080:00010000:3.0:1788572573.522452:0:111676:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dcb08) 00000080:00010000:3.0:1788572573.522456:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bda from it ffff9e0382023ac8 00010000:00010000:3.0:1788572573.522459:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.522473:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572573.522502:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572573.522524:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572573.522530:0:111676:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file1, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08), file ffff92fc49140f00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:3.0:1788572573.522537:0:111676:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file1, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08), intent=open|creat 00000100:00100000:3.0:1788572573.522616:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939200:192.168.202.132@tcp:900:bash.0 00000100:00100000:3.0:1788572573.522688:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.526482:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939200:192.168.202.132@tcp:900:bash.0 00000002:00010000:3.0:1788572573.526549:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 2l”vä¡6Èpë¥Òr¸,[0x200000422:0x2:0x0]) in obj [0x200000422:0x1:0x0], intent: open|creat flags 0101102 00010000:00010000:3.0:1788572573.526615:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 3/0,1 mode: --/CW res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.526626:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 3/0,1 mode: --/CW res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572573.526637:0:111676:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc48718800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572573.526646:0:111676:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6733b800 x1875449863939456/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 808/66432 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572573.526655:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 3/0,1 mode: --/CW res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572573.526676:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939456:192.168.202.132@tcp:101:bash.0 00000100:00100000:3.0:1788572573.527386:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc60988c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.531584:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939456:192.168.202.132@tcp:101:bash.0 00010000:00010000:3.0:1788572573.531613:0:111676:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 4/0,1 mode: --/CW res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531638:0:111676:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 4/0,1 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531645:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 4/0,1 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531656:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 4/0,1 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531667:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 4/0,1 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531679:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 5/1,1 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531687:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 5/1,1 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531695:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572573.531707:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff92fc6733b800 x1875449863939456/t4294983779(4294983779) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 808/696 e 0 to 0 dl 1788572589 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572573.531756:0:111676:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6733b800 x1875449863939456/t4294983779(4294983779) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 808/696 e 0 to 0 dl 1788572589 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572573.531772:0:111676:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572573.531786:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572573.531797:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 8589935650/2 (0 0) 00000080:00200000:3.0:1788572573.531803:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572573.531817:0:111676:0:(namei.c:1628:ll_create_it()) VFS Op:name=file1, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08), intent=open|creat 00000080:00200000:0.0:1788572573.532084:0:111676:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665ec2c8) 00000080:00200000:0.0:1788572573.532181:0:111676:0:(file.c:6843:ll_layout_conf()) [0x200000422:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572573.532187:0:111676:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000422:0x2:0x0](ffff92fc665ec2c8) 00000080:00010000:0.0:1788572573.532190:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665ec2c8) for lock 0x58a604dcef352be1 00000080:00010000:0.0:1788572573.532282:0:111676:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665ec2c8) 00000080:00200008:0.0:1788572573.532300:0:111676:0:(file.c:1090:ll_file_open()) START file file1:[0x200000422:0x2:0x0](ffff92fc665ec2c8), flags 101101 00000002:00100000:0.0:1788572573.532325:0:111676:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6733b800 x1875449863939456/t4294983779(4294983779) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572589 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572573.532372:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352be1 from it ffff92fc48215a20 00010000:00010000:0.0:1788572573.532377:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.532387:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572573.532410:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665ec2c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572573.532494:0:111676:0:(file.c:2650:do_file_write_iter()) START file file1:[0x200000422:0x2:0x0], ppos: 0, count: 2 00000002:00010000:0.0:1788572573.532520:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200000422:0x2:0x0], intent: getxattr flags 0 00010000:00010000:0.0:1788572573.532560:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.532568:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.532578:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572573.532599:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939584:192.168.202.132@tcp:101:bash.0 00000100:00100000:0.0:1788572573.532688:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.536323:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939584:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.536362:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.536379:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.536393:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572573.536408:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc6704b100 x1875449863939584/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 568/1104 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572573.536419:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.536433:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 8589935650/2 (0 0) 00000080:00200000:1.0:1788572573.536439:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x200000422:0x2:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788572573.536462:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665ec2c8) for lock 0x58a604dcef352be8 00000080:00010000:1.0:1788572573.536470:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352be8 from it ffff9e0382023af0 00010000:00010000:1.0:1788572573.536475:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.536484:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572573.536521:0:111676:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 2 bytes 00000080:00200000:1.0:1788572573.536529:0:111676:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 2 00000080:00200000:1.0:1788572573.536571:0:111676:0:(file.c:2090:ll_file_io_generic()) file1: write ppos: 0, bytes: 2 00000080:00200000:1.0:1788572573.536576:0:111676:0:(file.c:1848:ll_io_set_mirror()) file1: desiginated mirror: 0 00000020:00200000:1.0:1788572573.536584:0:111676:0:(cl_io.c:214:cl_io_rw_init()) header@00000000cd1e2f6a[0x110, 1, [0x200000422:0x2:0x0] hash] 00000020:00200000:1.0:1788572573.536586:0:111676:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 2) 0 0 00000080:00200000:1.0:1788572573.536591:0:111676:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572573.536605:0:111676:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:1.0:1788572573.536613:0:111676:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 2) 2, 2 00020000:00200000:1.0:1788572573.536617:0:111676:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572573.536647:0:111676:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:1.0:1788572573.536665:0:111676:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572573.536670:0:111676:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4fb0, osc lock ffff92fc47ff5430, flags 0x0 00010000:00010000:1.0:1788572573.536924:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.536935:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.536946:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,1 mode: --/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.536967:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939712:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.537212:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.541909:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939712:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.541949:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,1 mode: --/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.541969:0:111676:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.541988:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.542004:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572573.542016:0:111676:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,2 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.542031:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 5/0,2 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.542040:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788572573.542278:0:111676:0:(vvp_io.c:1292:vvp_io_write_start()) file1: write [0, 2) append 0 append lockpos 0 00000080:00200000:1.0:1788572573.542300:0:111676:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 2 bytes 00000080:00200000:1.0:1788572573.542372:0:111676:0:(rw26.c:871:ll_write_end()) pos 0, len 2, copied 2 00000080:00200000:1.0:1788572573.542393:0:111676:0:(rw26.c:929:ll_write_end()) page@00000000701e913b[3 00000000c088d5a1 2 1 00000000c443f556 0 0 0] 00000080:00200000:1.0:1788572573.542397:0:111676:0:(rw26.c:929:ll_write_end()) vmpage @00000000bed2faf9 17ffffc0001001 3:0 ffff92fc47cfe460 0 lru 00000080:00200000:1.0:1788572573.542404:0:111676:0:(rw26.c:929:ll_write_end()) osc-page@0000000082a2e091 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009f6c74d3 00000000f8b83fa8 > 3< 0 0 > 4< 0 0 8 8437760 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788572573.542406:0:111676:0:(rw26.c:929:ll_write_end()) end page@00000000701e913b 00000080:00200000:1.0:1788572573.542407:0:111676:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:1.0:1788572573.542412:0:111676:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 2 prio 0 00000008:00010000:1.0:1788572573.542434:0:111676:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000aafd8201 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,1 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788572573.542454:0:111676:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff92fc665ec478, count 1, dirtied 1 00000080:00200000:1.0:1788572573.542459:0:111676:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 2 bytes, tot: 2 00000080:00200000:1.0:1788572573.542462:0:111676:0:(vvp_io.c:1358:vvp_io_write_start()) file1: write bytes 0, result: 2 00000020:00010000:1.0:1788572573.542479:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f28b78 (ffff92fc49fce098/0) at cl_lock_release():245 00000020:00010000:1.0:1788572573.542482:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f28b78 (ffff92fc49fce098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572573.542485:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4fb0 (ffff92fc49fce3f8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572573.542491:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.542501:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788572573.542516:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f28b78 (ffff92fc49fce098/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572573.542519:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4fb0 (ffff92fc49fce098/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572573.542526:0:111676:0:(vvp_io.c:533:vvp_io_advance()) advancing 2 bytes 00000080:00200000:1.0:1788572573.542534:0:111676:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:1.0:1788572573.542545:0:111676:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572573.542548:0:111676:0:(file.c:2238:ll_file_io_generic()) file1: 2 io complete with rc: 0, result: 2, restart: 0 00000080:00200000:1.0:1788572573.542555:0:111676:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000422:0x2:0x0] updating i_size 2 i_blocks 0 00000080:00200000:1.0:1788572573.542560:0:111676:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 2 00000080:00200008:1.0:1788572573.542596:0:111676:0:(file.c:479:ll_file_release()) START file file1:[0x200000422:0x2:0x0](ffff92fc665ec2c8), flags 100001 00000002:00100000:1.0:1788572573.542664:0:111676:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6733b800 x1875449863939456/t4294983779(4294983779) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572589 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572573.542677:0:111676:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc48718800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572573.542683:0:111676:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67289500 x1875449863939968/t0(0) o35->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572573.542706:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939968:192.168.202.132@tcp:35:bash.0 00000100:00100000:1.0:1788572573.542977:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.545389:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863939968:192.168.202.132@tcp:35:bash.0 00000002:00100000:1.0:1788572573.545424:0:111676:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6733b800 x1875449863939456/t4294983779(4294983779) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572573.552654:0:111917:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:2.0:1788572573.552692:0:111917:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572573.552715:0:111917:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572573.552754:0:111917:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572573.552764:0:111917:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572573.552773:0:111917:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:2.0:1788572573.552777:0:111917:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352bd3 00000080:00010000:2.0:1788572573.552785:0:111917:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bd3 from it ffff9e03822fbbe0 00010000:00010000:2.0:1788572573.552790:0:111917:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572573.552799:0:111917:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572573.552818:0:111917:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572573.552839:0:111917:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60.sanity-sec, flags=24 00000080:00200000:2.0:1788572573.552846:0:111917:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572573.552858:0:111917:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08), flags=1536 00000080:00200000:2.0:1788572573.552861:0:111917:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572573.552866:0:111917:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572573.552871:0:111917:0:(namei.c:2172:do_mkdir()) VFS Op:name=subdir, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08) 00000100:00100000:2.0:1788572573.552963:0:111917:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111917:1875449863940096:192.168.202.132@tcp:700:mkdir.0 00000100:00100000:2.0:1788572573.553061:0:111917:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfa80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.555444:0:111917:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111917:1875449863940096:192.168.202.132@tcp:700:mkdir.0 00000020:00100000:2.0:1788572573.555520:0:111917:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc48718800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572573.555530:0:111917:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617e300 x1875449863940352/t0(0) o36->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 552/520 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572573.555550:0:111917:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111917:1875449863940352:192.168.202.132@tcp:36:mkdir.0 00000100:00100000:2.0:1788572573.555621:0:111917:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51ccfa80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.571412:0:111917:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job mkdir:8dd79b58-105c-49b4-a0c4-61b51a672039:111917:1875449863940352:192.168.202.132@tcp:36:mkdir.0 00000080:00200000:2.0:1788572573.571471:0:111917:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5348) 00000080:00200000:2.0:1788572573.571480:0:111917:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040b:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572573.599475:0:111676:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:1.0:1788572573.599503:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572573.599524:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572573.599541:0:111676:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572573.599549:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572573.599555:0:111676:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:1.0:1788572573.599559:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352bd3 00000080:00010000:1.0:1788572573.599566:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bd3 from it ffff9e0382023bf8 00010000:00010000:1.0:1788572573.599570:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572573.599584:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572573.599601:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572573.599608:0:111676:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d60.sanity-sec, flags=25 00000080:00200000:1.0:1788572573.599614:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572573.599624:0:111676:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08), flags=25 00000080:00200000:1.0:1788572573.599628:0:111676:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subdir, dir=[0x200000422:0x1:0x0](ffff92fc665dcb08), intent=0 00000002:00010000:1.0:1788572573.599654:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: Ì™C¸ æºî×Wè9f,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572573.599701:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.599709:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.599717:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.599997:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940480:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.600101:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.609653:0:111676:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc48718800: committing for last_committed 4294983780 gen 1 00000100:00100000:1.0:1788572573.609693:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940480:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.609714:0:111676:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609765:0:111676:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609775:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609786:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609798:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609811:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609819:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609837:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572573.609850:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc67288e00 x1875449863940480/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572573.609863:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.609873:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 9663677451/1 (0 0) 00000100:00100000:1.0:1788572573.609920:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940608:192.168.202.132@tcp:900:bash.0 00000100:00100000:1.0:1788572573.609985:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.612303:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940608:192.168.202.132@tcp:900:bash.0 00000002:00010000:1.0:1788572573.612360:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572573.612419:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.612429:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.612439:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.612458:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940736:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.612525:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.615199:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940736:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.615223:0:111676:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 4/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615235:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615249:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615263:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615276:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615290:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615297:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572573.615308:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff92fc67289500 x1875449863940736/t0(0) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572573.615319:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615342:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc42c63000 type 13 mode 4 res 9663677451/1 (0 0) 00000080:00200000:1.0:1788572573.615359:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572573.615380:0:111676:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040b:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572573.615384:0:111676:0:(namei.c:147:ll_iget()) got inode: [0x24000040b:0x1:0x0](ffff92fc665e5348): rc = 0 00000080:00010000:1.0:1788572573.615394:0:111676:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5348) for remote lock 0x58a604dcef352bfd 00000080:00010000:1.0:1788572573.615398:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5348) for lock 0x58a604dcef352bf6 00000080:00010000:1.0:1788572573.615408:0:111676:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5348) 00000080:00010000:1.0:1788572573.615413:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352bf6 from it ffff9e0382023ac8 00010000:00010000:1.0:1788572573.615417:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615425:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788572573.615440:0:111676:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x58a604dcef352bfd from it ffff9e0382023ac8 00010000:00010000:1.0:1788572573.615443:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615451:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572573.615470:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5348), inode mode 41ed mask 1 00000080:00200000:1.0:1788572573.615489:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5348), inode mode 41ed mask 3 00000080:00200000:1.0:1788572573.615494:0:111676:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=subfile1, dir=[0x24000040b:0x1:0x0](ffff92fc665e5348), file ffff92fc49140f00, open_flags 8241, mode 81b6 opened 0 00000080:00200000:1.0:1788572573.615502:0:111676:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subfile1, dir=[0x24000040b:0x1:0x0](ffff92fc665e5348), intent=open|creat 00000002:00010000:1.0:1788572573.615547:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: D¨æ5*XdQ>\K7„=¿,[0x24000040b:0x2:0x0]) in obj [0x24000040b:0x1:0x0], intent: open|creat flags 0101102 00010000:00010000:1.0:1788572573.615590:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 3/0,1 mode: --/CW res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.615597:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 3/0,1 mode: --/CW res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572573.615606:0:111676:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc48718800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572573.615612:0:111676:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67289500 x1875449863940992/t0(0) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 640/66264 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572573.615621:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 3/0,1 mode: --/CW res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.615636:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940992:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.615780:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.621293:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863940992:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.621320:0:111676:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 4/0,1 mode: --/CW res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621352:0:111676:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 4/0,1 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621362:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 4/0,1 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621373:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 4/0,1 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621385:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 4/0,1 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621399:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 5/1,1 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621408:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 5/1,1 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621416:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572573.621428:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff92fc67289500 x1875449863940992/t4294976772(4294976772) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 640/696 e 0 to 0 dl 1788572589 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572573.621441:0:111676:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67289500 x1875449863940992/t4294976772(4294976772) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 640/696 e 0 to 0 dl 1788572589 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572573.621455:0:111676:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572573.621469:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621480:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc42c63000 type 13 mode 4 res 9663677451/2 (0 0) 00000080:00200000:1.0:1788572573.621487:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x24000040b:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572573.621500:0:111676:0:(namei.c:1628:ll_create_it()) VFS Op:name=subfile1, dir=[0x24000040b:0x1:0x0](ffff92fc665e5348), intent=open|creat 00000080:00200000:1.0:1788572573.621529:0:111676:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d1148) 00000080:00200000:1.0:1788572573.621603:0:111676:0:(file.c:6843:ll_layout_conf()) [0x24000040b:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572573.621608:0:111676:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x24000040b:0x2:0x0](ffff92fc665d1148) 00000080:00010000:1.0:1788572573.621611:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d1148) for lock 0x58a604dcef352c04 00000080:00010000:1.0:1788572573.621688:0:111676:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d1148) 00000080:00200008:1.0:1788572573.621702:0:111676:0:(file.c:1090:ll_file_open()) START file subfile1:[0x24000040b:0x2:0x0](ffff92fc665d1148), flags 101101 00000002:00100000:1.0:1788572573.621754:0:111676:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67289500 x1875449863940992/t4294976772(4294976772) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572589 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572573.621789:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c04 from it ffff92fc5ddb7300 00010000:00010000:1.0:1788572573.621795:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.621805:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572573.622222:0:111676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d1148), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572573.622532:0:111676:0:(file.c:2650:do_file_write_iter()) START file subfile1:[0x24000040b:0x2:0x0], ppos: 0, count: 2 00000002:00010000:1.0:1788572573.622560:0:111676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040b:0x2:0x0], intent: getxattr flags 0 00010000:00010000:1.0:1788572573.622605:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.622616:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.622626:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 111676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.622650:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863941120:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.622810:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711ab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.626673:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863941120:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.626698:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.626714:0:111676:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.626792:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572573.626806:0:111676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc67288e00 x1875449863941120/t0(0) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 568/1112 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572573.626818:0:111676:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.626881:0:111676:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc42c63000 type 13 mode 4 res 9663677451/2 (0 0) 00000080:00200000:1.0:1788572573.626889:0:111676:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x24000040b:0x2:0x0] suppgids 0 -1: rc 0 00000080:00010000:1.0:1788572573.626914:0:111676:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d1148) for lock 0x58a604dcef352c0b 00000080:00010000:1.0:1788572573.626924:0:111676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c0b from it ffff9e0382023af0 00010000:00010000:1.0:1788572573.626928:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.626939:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572573.626970:0:111676:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 2 bytes 00000080:00200000:1.0:1788572573.626977:0:111676:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 2 00000080:00200000:1.0:1788572573.626988:0:111676:0:(file.c:2090:ll_file_io_generic()) subfile1: write ppos: 0, bytes: 2 00000080:00200000:1.0:1788572573.626993:0:111676:0:(file.c:1848:ll_io_set_mirror()) subfile1: desiginated mirror: 0 00000020:00200000:1.0:1788572573.627001:0:111676:0:(cl_io.c:214:cl_io_rw_init()) header@0000000060999004[0x110, 1, [0x24000040b:0x2:0x0] hash] 00000020:00200000:1.0:1788572573.627003:0:111676:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 2) 0 0 00000080:00200000:1.0:1788572573.627061:0:111676:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572573.627128:0:111676:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:1.0:1788572573.627136:0:111676:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 2) 2, 2 00020000:00200000:1.0:1788572573.627139:0:111676:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572573.627209:0:111676:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:1.0:1788572573.627223:0:111676:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572573.627227:0:111676:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4030, osc lock ffff92fc47ff5430, flags 0x0 00010000:00010000:1.0:1788572573.627269:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.627279:0:111676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.627354:0:111676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572573.627373:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863941248:192.168.202.132@tcp:101:bash.0 00000100:00100000:1.0:1788572573.627470:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711ab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572573.630881:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863941248:192.168.202.132@tcp:101:bash.0 00010000:00010000:1.0:1788572573.630908:0:111676:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.630926:0:111676:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.630937:0:111676:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.630950:0:111676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572573.630961:0:111676:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.630972:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.630980:0:111676:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788572573.631005:0:111676:0:(vvp_io.c:1292:vvp_io_write_start()) subfile1: write [0, 2) append 0 append lockpos 0 00000080:00200000:1.0:1788572573.631022:0:111676:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 2 bytes 00000080:00200000:1.0:1788572573.631134:0:111676:0:(rw26.c:871:ll_write_end()) pos 0, len 2, copied 2 00000080:00200000:1.0:1788572573.631145:0:111676:0:(rw26.c:929:ll_write_end()) page@000000006daf59d2[3 000000005b08b836 2 1 00000000c443f556 0 0 0] 00000080:00200000:1.0:1788572573.631215:0:111676:0:(rw26.c:929:ll_write_end()) vmpage @000000006f4437c4 17ffffc0001001 3:0 ffff92fc47cfe1c0 0 lru 00000080:00200000:1.0:1788572573.631224:0:111676:0:(rw26.c:929:ll_write_end()) osc-page@00000000460e2240 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009f6c74d3 0000000007ece5f1 > 3< 0 0 > 4< 0 0 8 8409088 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:1.0:1788572573.631226:0:111676:0:(rw26.c:929:ll_write_end()) end page@000000006daf59d2 00000080:00200000:1.0:1788572573.631228:0:111676:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:1.0:1788572573.631232:0:111676:0:(vvp_io.c:1175:vvp_io_write_commit()) commit async pages: 1, from 0, to 2 prio 0 00000008:00010000:1.0:1788572573.631252:0:111676:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000c40fa5d2 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788572573.631271:0:111676:0:(vvp_io.c:1088:vvp_set_batch_dirty()) mapping ffff92fc665d12f8, count 1, dirtied 1 00000080:00200000:1.0:1788572573.631276:0:111676:0:(vvp_io.c:1198:vvp_io_write_commit()) Committed 1 pages 2 bytes, tot: 2 00000080:00200000:1.0:1788572573.631279:0:111676:0:(vvp_io.c:1358:vvp_io_write_start()) subfile1: write bytes 0, result: 2 00000020:00010000:1.0:1788572573.631370:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f28b78 (ffff92fc49fce098/0) at cl_lock_release():245 00000020:00010000:1.0:1788572573.631374:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f28b78 (ffff92fc49fce098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572573.631378:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4030 (ffff92fc49fce3f8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1788572573.631383:0:111676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572573.631393:0:111676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:1.0:1788572573.631407:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f28b78 (ffff92fc49fce098/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572573.631410:0:111676:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4030 (ffff92fc49fce098/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572573.631418:0:111676:0:(vvp_io.c:533:vvp_io_advance()) advancing 2 bytes 00000080:00200000:1.0:1788572573.631425:0:111676:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:1.0:1788572573.631433:0:111676:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572573.631437:0:111676:0:(file.c:2238:ll_file_io_generic()) subfile1: 2 io complete with rc: 0, result: 2, restart: 0 00000080:00200000:1.0:1788572573.631444:0:111676:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040b:0x2:0x0] updating i_size 2 i_blocks 0 00000080:00200000:1.0:1788572573.631449:0:111676:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 2 00000080:00200008:1.0:1788572573.631484:0:111676:0:(file.c:479:ll_file_release()) START file subfile1:[0x24000040b:0x2:0x0](ffff92fc665d1148), flags 100001 00000002:00100000:1.0:1788572573.631545:0:111676:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67289500 x1875449863940992/t4294976772(4294976772) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572589 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572573.631559:0:111676:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc48718800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572573.631565:0:111676:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67288e00 x1875449863941504/t0(0) o35->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572573.631589:0:111676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863941504:192.168.202.132@tcp:35:bash.0 00000100:00100000:1.0:1788572573.631743:0:111676:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711ab40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.634799:0:111676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job bash:8dd79b58-105c-49b4-a0c4-61b51a672039:111676:1875449863941504:192.168.202.132@tcp:35:bash.0 00000002:00100000:2.0:1788572573.634844:0:111676:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67289500 x1875449863940992/t4294976772(4294976772) o101->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 624/696 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788572573.645052:0:111921:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:2.0:1788572573.645247:0:111921:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc48718800 00010000:00010000:2.0:1788572573.645410:0:111921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 5/5 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788572573.645452:0:111877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 5 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572573.645462:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.645504:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645518:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645537:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572573.645552:0:111877:0:(vvp_object.c:117:vvp_conf_set()) [0x200000422:0x2:0x0]: losing layout lock 00010000:00010000:0.0:1788572573.645572:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645615:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572573.645624:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645632:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 1/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645638:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 1/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645645:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 1/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.645651:0:111877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 5 locks packed 00000100:00100000:0.0:1788572573.645679:0:111877:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:8dd79b58-105c-49b4-a0c4-61b51a672039:111877:1875449863941632:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572573.646277:0:111877:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53d4ad80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572573.649386:0:111877:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:8dd79b58-105c-49b4-a0c4-61b51a672039:111877:1875449863941632:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572573.649418:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4600/0x58a604dcef352bd3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217930ed expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572573.649444:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4400/0x58a604dcef352bf6 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793180 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572573.649457:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc51bc2800/0x58a604dcef352be8 lrc: 0/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179313a expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572573.649465:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205400/0x58a604dcef352be1 lrc: 0/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a221793125 expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572573.649476:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc70205000/0x58a604dcef352bda lrc: 0/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793109 expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572573.649503:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.649505:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.649508:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572573.649773:0:111921:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc48718800 00010000:00010000:0.0:1788572573.649791:0:111921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572573.649972:0:111876:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572573.649981:0:111876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.650012:0:111876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572573.650034:0:111876:0:(vvp_object.c:117:vvp_conf_set()) [0x24000040b:0x2:0x0]: losing layout lock 00010000:00010000:1.0:1788572573.650068:0:111876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.650103:0:111876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.650111:0:111876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.650118:0:111876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.650125:0:111876:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:1.0:1788572573.650146:0:111876:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:8dd79b58-105c-49b4-a0c4-61b51a672039:111876:1875449863941760:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572573.650229:0:111876:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711acc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572573.654774:0:111876:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:8dd79b58-105c-49b4-a0c4-61b51a672039:111876:1875449863941760:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572573.654807:0:111876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4800/0x58a604dcef352c0b lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217931bf expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.654834:0:111876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f5e00/0x58a604dcef352c04 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217931aa expref: -99 pid: 111676 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.654852:0:111876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc48718800 lock: ffff92fc702f4e00/0x58a604dcef352bfd lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179318e expref: -99 pid: 111676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572573.654871:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572573.654873:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572573.654875:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.655085:0:111921:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc48718800 00010000:00010000:2.0:1788572573.655099:0:111921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572573.655386:0:111877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572573.655394:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788572573.655419:0:111877:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000c40fa5d2 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788572573.655431:0:111877:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000c40fa5d2 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.655480:0:111877:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:1.0:1788572573.655594:0:111877:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc67288000 - object 0x280000400:24 offset 0<>4096 00000100:00100000:1.0:1788572573.655655:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572573.655671:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8dd79b58-105c-49b4-a0c4-61b51a672039:2393:1875449863941888:192.168.202.132@tcp:4:bash.0 00000008:00010000:0.0:1788572573.711109:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000c40fa5d2 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788572573.711187:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000c40fa5d2 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572573.711215:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8dd79b58-105c-49b4-a0c4-61b51a672039:2393:1875449863941888:192.168.202.132@tcp:4:bash.0 00000080:00200000:3.0:1788572573.711261:0:111877:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572573.711342:0:111877:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1788572573.711370:0:111877:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788572573.711390:0:111877:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000aafd8201 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788572573.711403:0:111877:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000aafd8201 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572573.711444:0:111877:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 4/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:3.0:1788572573.711549:0:111877:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc6733aa00 - object 0x280000401:553 offset 0<>4096 00000100:00100000:3.0:1788572573.711608:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8dd79b58-105c-49b4-a0c4-61b51a672039:2396:1875449863942016:192.168.202.132@tcp:4:bash.0 00000008:00010000:3.0:1788572573.726341:0:2396:0:(osc_cache.c:908:osc_extent_finish()) ### extent: 00000000aafd8201 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:3.0:1788572573.726407:0:2396:0:(osc_cache.c:497:__osc_extent_remove()) ### extent: 00000000aafd8201 ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400000000 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788572573.726438:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8dd79b58-105c-49b4-a0c4-61b51a672039:2396:1875449863942016:192.168.202.132@tcp:4:bash.0 00000080:00200000:1.0:1788572573.726749:0:111877:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572573.726823:0:111877:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788572573.726884:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 1/0,0 mode: --/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.726895:0:111877:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 1/0,0 mode: --/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572573.726902:0:111877:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:1.0:1788572573.726922:0:111877:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:8dd79b58-105c-49b4-a0c4-61b51a672039:111877:1875449863942144:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572573.726993:0:111877:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711ac00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572573.731560:0:111877:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:8dd79b58-105c-49b4-a0c4-61b51a672039:111877:1875449863942144:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572573.731610:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc48718800 lock: 00000000c1d3ffa1/0x58a604dcef352c12 lrc: 0/0,0 mode: --/PW res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0xc9a239a2217931c6 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572573.731642:0:111877:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc48718800 lock: 000000005315e325/0x58a604dcef352bef lrc: 0/0,0 mode: --/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0xc9a239a221793141 expref: -99 pid: 111676 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572573.731673:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572573.731676:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572573.731678:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572573.731901:0:111921:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc48718800 00000080:00200000:2.0:1788572573.794291:0:40043:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572573.794312:0:40043:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572573.794414:0:40043:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572573.794430:0:40043:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572573.794435:0:40043:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572573.794443:0:40043:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572573.904103:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5348) 00000080:00200000:0.0:1788572573.904154:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dcb08) 00000080:00200000:0.0:1788572573.904180:0:111676:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572573.904193:0:111676:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572573.904256:0:111676:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572573.904262:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d1148) 00000080:00200000:0.0:1788572573.904391:0:111676:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572573.904397:0:111676:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572573.904405:0:111676:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572573.904410:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665ec2c8) 00010000:00010000:3.0:1788572574.084951:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572574.992049:0:111938:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665ea1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572574.992065:0:111938:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:1.0:1788572574.992089:0:111938:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572574.992312:0:111938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 111938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572574.992324:0:111938:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111938 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572574.992336:0:111938:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 111938 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572574.992363:0:111938:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job lsof:8dd79b58-105c-49b4-a0c4-61b51a672039:111938:1875449863942272:192.168.202.132@tcp:101:lsof.0 00000100:00100000:1.0:1788572574.992465:0:111938:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711aa80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572574.999865:0:111938:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job lsof:8dd79b58-105c-49b4-a0c4-61b51a672039:111938:1875449863942272:192.168.202.132@tcp:101:lsof.0 00010000:00010000:0.0:1788572574.999897:0:111938:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572574.999911:0:111938:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572574.999924:0:111938:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572574.999935:0:111938:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572574.999947:0:111938:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572574.999954:0:111938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572574.999978:0:111938:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572574.999989:0:111938:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b800 x1875449863942272/t0(0) o101->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572590 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572575.000004:0:111938:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.000224:0:111938:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc8206b400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788572575.000234:0:111938:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572575.000270:0:111938:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1788572575.000287:0:111938:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352c19 00000080:00010000:0.0:1788572575.000293:0:111938:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:0.0:1788572575.000297:0:111938:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c19 from it ffff9e0382343cb8 00010000:00010000:0.0:1788572575.000300:0:111938:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572575.000310:0:111938:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572575.109116:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572576.145856:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572577.166256:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572578.181106:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788572578.416158:0:111946:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665ea1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572578.416173:0:111946:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:0.0:1788572578.416197:0:111946:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572578.416217:0:111946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572578.416302:0:111946:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572578.416313:0:111946:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572578.416322:0:111946:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:0.0:1788572578.416326:0:111946:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352c19 00000080:00010000:0.0:1788572578.416334:0:111946:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c19 from it ffff9e0382163cb8 00010000:00010000:0.0:1788572578.416338:0:111946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572578.416349:0:111946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:0.0:1788572578.416511:0:111946:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff92fc665ea1c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572578.416516:0:111946:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8),name=/ 00000002:00010000:0.0:1788572578.416524:0:111946:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572578.416532:0:111946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572578.416542:0:111946:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572578.416549:0:111946:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572578.416553:0:111946:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00010000:0.0:1788572578.416556:0:111946:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff92fc665ea1c8) for lock 0x58a604dcef352c19 00000080:00010000:0.0:1788572578.416562:0:111946:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c19 from it ffff9e0382163cb8 00010000:00010000:0.0:1788572578.416566:0:111946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572578.416573:0:111946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572578.435445:0:111946:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff92fc665ea1c8) 00000080:00200000:0.0:1788572578.435493:0:111946:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc48718800 (sb ffff92fc48718800) 00010000:00010000:0.0:1788572578.435538:0:111946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435551:0:111946:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435558:0:111946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435565:0:111946:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435598:0:111946:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788572578.435605:0:111946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435613:0:111946:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435620:0:111946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435628:0:111946:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435654:0:111946:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1788572578.435667:0:111876:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788572578.435674:0:111876:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435681:0:111876:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3e00 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572578.435686:0:111876:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435696:0:111876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572578.435704:0:111876:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572578.435711:0:111876:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572578.435714:0:111876:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 00010000:00010000:0.0:1788572578.435782:0:111876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435789:0:111876:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572578.435812:0:111876:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435820:0:111876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc3e00/0x58a604dcef352b63 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793092 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435839:0:111876:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572578.435843:0:111876:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435849:0:111876:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2600 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572578.435852:0:111876:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435859:0:111876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572578.435866:0:111876:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572578.435872:0:111876:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572578.435874:0:111876:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:0.0:1788572578.435876:0:111876:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:0.0:1788572578.435888:0:111876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435894:0:111876:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572578.435899:0:111876:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435907:0:111876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2600/0x58a604dcef352b94 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a221793099 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.435917:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572578.435919:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572578.435921:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572578.435943:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c727f535-e4fd-4a83-8271-2859b65ffa9f:2393:1875449863942400:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572578.436051:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572578.436059:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c727f535-e4fd-4a83-8271-2859b65ffa9f:2393:1875449863942528:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572578.436225:0:111946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436234:0:111946:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436240:0:111946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436247:0:111946:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436260:0:111946:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:01000000:0.0:1788572578.436264:0:111946:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:0.0:1788572578.436269:0:111946:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:0.0:1788572578.436289:0:111876:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572578.436293:0:111876:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436299:0:111876:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70366a00 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572578.436302:0:111876:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436309:0:111876:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572578.436318:0:111876:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572578.436323:0:111876:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:0.0:1788572578.436325:0:111876:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:0.0:1788572578.436327:0:111876:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:0.0:1788572578.436341:0:111876:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436346:0:111876:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572578.436356:0:111876:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436361:0:111876:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352b9b lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217930a0 expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572578.436372:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572578.436373:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572578.436375:0:111876:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572578.436380:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788572578.436389:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c727f535-e4fd-4a83-8271-2859b65ffa9f:2394:1875449863942656:192.168.202.132@tcp:103:ldlm_bl.0 00020000:01000000:0.0:1788572578.436967:0:111946:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc48dc3800 active: 1 00020000:01000000:0.0:1788572578.436973:0:111946:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc48dc0000 active: 1 00020000:01000000:0.0:1788572578.436976:0:111946:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:0.0:1788572578.436981:0:111946:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc48718800 00020000:01000000:0.0:1788572578.436983:0:111946:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc48718800: disconnecting target lustre-OST0001-osc-ffff92fc48718800 00000100:00080000:0.0:1788572578.436995:0:111946:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-OST0001_UUID 00000100:00080000:0.0:1788572578.437097:0:111946:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc6000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1788572578.441598:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863942784:192.168.202.132@tcp:9:umount.0 00010000:00010000:1.0:1788572578.443755:0:111877:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572578.446086:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c727f535-e4fd-4a83-8271-2859b65ffa9f:2393:1875449863942528:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572578.446103:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c727f535-e4fd-4a83-8271-2859b65ffa9f:2393:1875449863942400:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572578.450194:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c727f535-e4fd-4a83-8271-2859b65ffa9f:2394:1875449863942656:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:0.0:1788572578.450256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc6000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:0.0:1788572578.450278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863942784:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572578.450377:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788572578.450381:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc48718800: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572578.450387:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572578.450520:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352bc5 00000020:00000080:1.0:1788572578.450538:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc48dc0000/LOV_OSC_UUID 00000020:01000000:1.0:1788572578.450594:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc48718800 (flags='') 00000020:00000080:1.0:1788572578.450600:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572578.450609:0:111946:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc48718800: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572578.450613:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc700319c0) has no exports 00000020:00000080:0.0:1788572578.451536:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc0000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc48718800 00000008:01000000:2.0:1788572578.451808:0:111946:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc48718800: client import never connected 00000100:00080000:2.0:1788572578.451813:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:2.0:1788572578.451816:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc48718800: committing for last_committed 0 gen 3 00000100:00100000:2.0:1788572578.451821:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572578.451994:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572578.452003:0:111946:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00020000:01000000:2.0:1788572578.452007:0:111946:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc48718800 00020000:01000000:2.0:1788572578.452009:0:111946:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc48718800: disconnecting target lustre-OST0000-osc-ffff92fc48718800 00000100:00080000:2.0:1788572578.452026:0:111946:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-OST0000_UUID 00000100:00080000:2.0:1788572578.452130:0:111946:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:0.0:1788572578.452288:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc48dc6000 for lustre-OST0001-osc-ffff92fc48718800 00000020:00000080:0.0:1788572578.452299:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc48dc3000/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:0.0:1788572578.452301:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc3000/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-OST0001-osc-ffff92fc48718800 00000020:01000000:0.0:1788572578.452307:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc48718800 (8dd79b58-105c-49b4-a0c4-61b51a672039) 00000100:00100000:0.0:1788572578.452512:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863942912:192.168.202.132@tcp:9:umount.0 00000100:00080000:0.0:1788572578.481340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5000 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:0.0:1788572578.481371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863942912:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572578.481769:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572578.481773:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc48718800: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788572578.481783:0:111946:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67288000 x1875449863941888/t4294968536(4294968536) o4->lustre-OST0000-osc-ffff92fc48718800@192.168.202.132@tcp:6/4 lens 4584/448 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572578.481807:0:111946:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc6733aa00 x1875449863942016/t4294968537(4294968537) o4->lustre-OST0000-osc-ffff92fc48718800@192.168.202.132@tcp:6/4 lens 4584/448 e 0 to 0 dl 1788572589 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572578.481820:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572578.481977:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352bbe 00000020:00000080:1.0:1788572578.481993:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc48dc3800/LOV_OSC_UUID 00000020:01000000:1.0:1788572578.482036:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc48718800 (flags='') 00000020:00000080:1.0:1788572578.482042:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572578.482050:0:111946:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc48718800: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572578.482054:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc70034d40) has no exports 00000020:00000080:0.0:1788572578.485577:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc3800/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc48718800 00000008:01000000:1.0:1788572578.486338:0:111946:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc48718800: client import never connected 00000100:00080000:1.0:1788572578.486344:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788572578.486349:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc48718800: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572578.486356:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572578.487501:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572578.487515:0:111946:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:00000080:1.0:1788572578.487521:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352bb7 00000020:00000080:1.0:1788572578.487530:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc48dc7000/8dd79b58-105c-49b4-a0c4-61b51a672039 00000100:00080000:1.0:1788572578.487612:0:111946:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-MDT0000_UUID 00000020:00000080:0.0:1788572578.487807:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc48dc5000 for lustre-OST0000-osc-ffff92fc48718800 00000020:00000080:0.0:1788572578.487815:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc48dc2800/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:0.0:1788572578.487818:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc2800/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-OST0000-osc-ffff92fc48718800 00000020:01000000:0.0:1788572578.487824:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc48718800 (8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:00000080:0.0:1788572578.488070:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc7000/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-clilov-ffff92fc48718800 00010000:00010000:1.0:1788572578.488813:0:111946:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572578.488867:0:111946:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572578.488875:0:111946:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788572578.488893:0:111946:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc6711a900 going to sleep for 0 seconds 00000100:00100000:1.0:1788572578.488904:0:111946:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job umount:8dd79b58-105c-49b4-a0c4-61b51a672039:111946:1875449863943040:192.168.202.132@tcp:103:umount.0 00010000:00010000:1.0:1788572578.488915:0:111946:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788572578.488924:0:111946:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788572578.488928:0:111946:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc48718800 lock: ffff92fc702f4c00/0x58a604dcef352c19 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217931d4 expref: -99 pid: 111938 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00080000:1.0:1788572578.494295:0:111946:0:(import.c:65:import_set_state_nolock()) ffff92fc51bec800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1788572578.494380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863943168:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572578.513909:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc51bec800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572578.514052:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863943168:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572578.514913:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572578.514919:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc48718800: committing for last_committed 4294983780 gen 2 00000100:00100000:1.0:1788572578.514931:0:111946:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67288000 x1875449863943168/t4294983782(4294983782) o39->lustre-MDT0000-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572588 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572578.514957:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572578.516808:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352ba9 00000020:00000080:1.0:1788572578.516825:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc50c38000/8dd79b58-105c-49b4-a0c4-61b51a672039 00000100:00080000:1.0:1788572578.516912:0:111946:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 8dd79b58-105c-49b4-a0c4-61b51a672039->lustre-MDT0001_UUID 00000100:00080000:1.0:1788572578.518088:0:111946:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc1000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572578.518917:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863943296:192.168.202.132@tcp:39:umount.0 00000020:00000080:2.0:1788572578.519120:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc50c38000/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-MDT0000-mdc-ffff92fc48718800 00000100:00100000:3.0:1788572578.534562:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc48718800: committing for last_committed 4294976773 gen 1 00000100:00100000:3.0:1788572578.534607:0:2392:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff92fc67288000 x1875449863943296/t4294976774(4294976774) o39->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572588 ref 2 fl Rpc:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572578.534628:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc1000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572578.534659:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8dd79b58-105c-49b4-a0c4-61b51a672039:2392:1875449863943296:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572578.534813:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572578.534818:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc48718800: committing for last_committed 4294976773 gen 2 00000100:00100000:1.0:1788572578.534826:0:111946:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67288000 x1875449863943296/t4294976774(4294976774) o39->lustre-MDT0001-mdc-ffff92fc48718800@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572588 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572578.534842:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572578.540384:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352bb0 00000020:00000080:1.0:1788572578.540400:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc5838c800/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:2.0:1788572578.542217:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc5838c800/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-MDT0001-mdc-ffff92fc48718800 00000020:00000080:2.0:1788572578.542271:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352ba2 00000020:00000080:2.0:1788572578.542280:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc481c2800/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:01000000:2.0:1788572578.542306:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc48718800 (flags='') 00000020:00000080:2.0:1788572578.542315:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572578.542321:0:111946:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc48718800: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572578.542326:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc48cc99c0) has no exports 00000020:00000080:2.0:1788572578.542390:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572578.542397:0:111946:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:00000080:2.0:1788572578.542400:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc51bee800/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:2.0:1788572578.542403:0:111946:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51bee800/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-clilov-ffff92fc48718800 00000020:01000000:2.0:1788572578.542407:0:111946:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc48718800 (8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:2.0:1788572578.542558:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc48718800 (flags='') 00000020:00000080:2.0:1788572578.542561:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572578.542565:0:111946:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc48718800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572578.542567:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc48ccb380) has no exports 00000020:00000080:2.0:1788572578.542603:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572578.542607:0:111946:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:2.0:1788572578.542610:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc48718800 (flags='') 00000020:00000080:2.0:1788572578.542613:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572578.542615:0:111946:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc48718800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572578.542619:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc48cccd40) has no exports 00000020:00000080:0.0:1788572578.542886:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc481c2800/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-clilmv-ffff92fc48718800 00000020:00000080:0.0:1788572578.542891:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc51be8800/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:0.0:1788572578.542893:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51be8800/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-clilmv-ffff92fc48718800 00000020:01000000:0.0:1788572578.542900:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc48718800 (8dd79b58-105c-49b4-a0c4-61b51a672039) 00000008:01000000:2.0:1788572578.543192:0:111946:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc48718800: client import never connected 00000100:00080000:2.0:1788572578.543198:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572578.543204:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc48718800: committing for last_committed 4294983780 gen 3 00000100:00100000:2.0:1788572578.543211:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572578.549211:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572578.549224:0:111946:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000020:01000000:2.0:1788572578.549230:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc48718800 (flags='') 00000020:00000080:2.0:1788572578.549234:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572578.549238:0:111946:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc48718800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572578.549243:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc70033380) has no exports 00000020:00000080:0.0:1788572578.549578:0:110631:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc51bec800 for lustre-MDT0000-mdc-ffff92fc48718800 00000020:00000080:0.0:1788572578.549587:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc51be9000/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:0.0:1788572578.549590:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51be9000/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-MDT0000-mdc-ffff92fc48718800 00000020:01000000:0.0:1788572578.549595:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc48718800 (8dd79b58-105c-49b4-a0c4-61b51a672039) 00000008:01000000:2.0:1788572578.550417:0:111946:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc48718800: client import never connected 00000100:00080000:2.0:1788572578.550424:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:2.0:1788572578.550427:0:111946:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc48718800: committing for last_committed 4294976773 gen 3 00000100:00100000:2.0:1788572578.550435:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:2.0:1788572578.559969:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572578.559986:0:111946:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc48718800 (uuid 8dd79b58-105c-49b4-a0c4-61b51a672039) 00000080:02000400:2.0:1788572578.559992:0:111946:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:2.0:1788572578.559994:0:111946:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:2.0:1788572578.560043:0:111946:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc48718800 00000100:00080000:2.0:1788572578.560053:0:111946:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import c727f535-e4fd-4a83-8271-2859b65ffa9f->MGS 00010000:00010000:2.0:1788572578.560068:0:111946:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572578.560082:0:111946:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572578.560088:0:111946:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572578.560091:0:111946:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:2.0:1788572578.560094:0:111946:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:2.0:1788572578.560148:0:111946:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572578.560155:0:111946:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788572578.560173:0:111946:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45182840 going to sleep for 0 seconds 00000100:00100000:2.0:1788572578.560184:0:111946:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job umount:c727f535-e4fd-4a83-8271-2859b65ffa9f:111946:1875449863943424:192.168.202.132@tcp:103:umount.0 00010000:00010000:2.0:1788572578.560195:0:111946:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:2.0:1788572578.560202:0:111946:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:2.0:1788572578.560206:0:111946:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc51bc2200/0x58a604dcef352b5c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a22179308b expref: -99 pid: 111865 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572578.560230:0:111946:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0f800 MGS: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572578.560256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c727f535-e4fd-4a83-8271-2859b65ffa9f:2392:1875449863943552:192.168.202.132@tcp:251:umount.0 00000020:00000080:3.0:1788572578.563599:0:110631:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc48dc1000 for lustre-MDT0001-mdc-ffff92fc48718800 00000020:00000080:3.0:1788572578.563623:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc48dc0800/8dd79b58-105c-49b4-a0c4-61b51a672039 00000020:00000080:3.0:1788572578.563627:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc0800/8dd79b58-105c-49b4-a0c4-61b51a672039 for lustre-MDT0001-mdc-ffff92fc48718800 00000020:01000000:3.0:1788572578.563641:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc48718800 (8dd79b58-105c-49b4-a0c4-61b51a672039) 00000100:00080000:2.0:1788572578.596813:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0f800 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:2.0:1788572578.596825:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572578.596829:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572578.596830:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572578.596843:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c727f535-e4fd-4a83-8271-2859b65ffa9f:2392:1875449863943552:192.168.202.132@tcp:251:umount.0 00000100:00080000:2.0:1788572578.596914:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572578.596917:0:111946:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572578.596920:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572578.596922:0:111946:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572578.596933:0:111946:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352b55 00000020:00000080:2.0:1788572578.596943:0:111946:0:(genops.c:885:class_export_put()) final put ffff92fc492a5000/c727f535-e4fd-4a83-8271-2859b65ffa9f 00000020:01000000:2.0:1788572578.596967:0:111946:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:2.0:1788572578.596974:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572578.596982:0:111946:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572578.596986:0:111946:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc48cc8000) has no exports 00000020:00000080:2.0:1788572578.599429:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc492a5000/c727f535-e4fd-4a83-8271-2859b65ffa9f for MGC192.168.202.132@tcp 10000000:01000000:1.0:1788572578.602781:0:111879:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 10000000:01000000:2.0:1788572578.608209:0:111946:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:2.0:1788572578.608219:0:111946:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572578.608224:0:111946:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572578.608229:0:111946:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572578.608232:0:111946:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572578.608324:0:111946:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572578.608332:0:111946:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid c727f535-e4fd-4a83-8271-2859b65ffa9f) 00000020:01000004:2.0:1788572578.608338:0:111946:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc48718800 1 00000020:01000004:2.0:1788572578.608342:0:111946:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc48719000 00000020:00000080:1.0:1788572578.608587:0:110631:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc42c0f800 for MGC192.168.202.132@tcp 00000020:00000080:1.0:1788572578.608596:0:110631:0:(genops.c:885:class_export_put()) final put ffff92fc4871c800/c727f535-e4fd-4a83-8271-2859b65ffa9f 00000020:00000080:1.0:1788572578.608599:0:110631:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4871c800/c727f535-e4fd-4a83-8271-2859b65ffa9f for MGC192.168.202.132@tcp 00000020:01000000:1.0:1788572578.608608:0:110631:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (c727f535-e4fd-4a83-8271-2859b65ffa9f) 00010000:00010000:1.0:1788572578.611120:0:111876:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:1.0:1788572578.611343:0:111877:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00000100:00100000:0.0:1788572578.617496:0:111866:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc483d6400:111866] 0 exiting: rc = 1 00000100:00100000:0.0:1788572578.617601:0:111867:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc51f22600:111867] 1 exiting: rc = 0 00000100:00100000:2.0:1788572578.624273:0:111868:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc51f23e00:111868] 0 exiting: rc = 1 00000100:00100000:2.0:1788572578.624368:0:111869:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff92fc51201000:111869] 1 exiting: rc = 0 00000100:00100000:1.0:1788572578.626439:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.626452:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.627900:0:111871:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc51f23400:111871] 1 exiting: rc = 0 00000100:00100000:0.0:1788572578.629049:0:111870:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc51200000:111870] 0 exiting: rc = 1 00000100:00100000:0.0:1788572578.629167:0:111872:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc51201a00:111872] 2 exiting: rc = 0 00000100:00100000:2.0:1788572578.632598:0:111873:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc491ec000:111873] 0 exiting: rc = 1 00000100:00100000:2.0:1788572578.632815:0:111874:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc49da6000:111874] 1 exiting: rc = 0 00000100:00100000:2.0:1788572578.632869:0:111875:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc49da6a00:111875] 2 exiting: rc = 0 00000100:00100000:1.0:1788572578.633792:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633802:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633807:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633812:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633819:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633824:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633831:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633837:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633842:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633847:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633852:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633859:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633864:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633870:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633875:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633881:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633886:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633892:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633897:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633903:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633910:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633916:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633922:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633927:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633932:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633937:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633942:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633948:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633953:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633958:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633963:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633968:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633974:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633980:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633985:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633990:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.633995:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634000:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634005:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634010:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634014:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634020:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634025:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634030:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634035:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634039:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634045:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634049:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634054:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634058:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634064:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634068:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634073:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634079:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634084:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634089:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634094:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634099:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634103:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634109:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634114:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634119:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634124:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634129:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634148:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634154:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634161:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634167:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634172:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634177:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634183:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634189:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634194:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634199:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634205:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634210:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634215:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634221:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634226:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634231:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634236:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634242:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634248:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634254:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634260:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634265:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634271:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634276:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634282:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634287:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634292:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634298:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634302:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634307:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634313:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634318:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634323:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634329:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634335:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634341:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634346:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634351:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634356:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634363:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634367:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634373:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634379:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634385:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634390:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634396:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634401:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634406:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634412:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634418:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634423:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634428:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634433:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634438:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634444:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634450:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634455:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634460:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634466:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634471:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634477:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634483:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634488:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572578.634493:0:110631:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:2.0:1788572579.462194:0:111965:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:2.0:1788572579.462198:0:111965:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:2.0:1788572579.462200:0:111965:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre/d60.sanity-sec 00000020:01000004:2.0:1788572579.462201:0:111965:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:2.0:1788572579.462310:0:111965:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc704fb000 00000080:01000004:2.0:1788572579.462317:0:111965:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:2.0:1788572579.462332:0:111965:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:2.0:1788572579.462357:0:111965:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:2.0:1788572579.462368:0:111965:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc48cc99c0) 00000020:00000080:2.0:1788572579.463042:0:111965:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:2.0:1788572579.463298:0:111965:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788572579.463323:0:111965:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788572579.463533:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:2.0:1788572579.463540:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572579.464489:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572579.464497:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:2.0:1788572579.465530:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:1.0:1788572579.465593:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572579.469445:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:1.0:1788572579.469460:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572579.470718:0:111965:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572579.471438:0:111965:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572579.471712:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572579.471820:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:1.0:1788572579.472704:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:1.0:1788572579.472714:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:2.0:1788572579.480014:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:2.0:1788572579.480026:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:2.0:1788572579.480473:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:2.0:1788572579.480480:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:3.0:1788572579.482290:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:3.0:1788572579.482322:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:1.0:1788572579.483293:0:111965:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:1.0:1788572579.483303:0:111965:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:0.0:1788572579.484369:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572579.484435:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572579.484437:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572579.484822:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572579.484824:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572579.484827:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788572579.485505:0:111965:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc42c0c000@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:1.0:1788572579.485807:0:111965:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:1.0:1788572579.485816:0:111965:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:1.0:1788572579.488024:0:111980:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:1.0:1788572579.488134:0:111965:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid c95aecce-2623-4667-b83f-7aa0f7bacb74) 00000020:00000080:1.0:1788572579.488159:0:111965:0:(genops.c:1385:class_connect()) connect: client c95aecce-2623-4667-b83f-7aa0f7bacb74, cookie 0x58a604dcef352c27 00000100:00080000:1.0:1788572579.488175:0:111965:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572579.488186:0:111965:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc42c0c000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572579.488220:0:111965:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449863943680/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572579.488243:0:111965:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import c95aecce-2623-4667-b83f-7aa0f7bacb74->MGS 00000080:00200000:1.0:1788572579.488269:0:111965:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc704fb000 (sb ffff92fc704fb000) 00000080:01000000:1.0:1788572579.488325:0:111965:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:1.0:1788572579.488328:0:111965:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:1.0:1788572579.488332:0:111965:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:1.0:1788572579.488339:0:111965:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 802259b2-6b09-4a33-8d4c-554d26b1ec81 00010000:00010000:0.0:1788572579.488833:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572579.489791:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c95aecce-2623-4667-b83f-7aa0f7bacb74:2392:1875449863943680:192.168.202.132@tcp:250:mount.lustre.0 10000000:01000000:1.0:1788572579.490279:0:111965:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:1.0:1788572579.490283:0:111965:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc704fb000 10000000:01000000:1.0:1788572579.490288:0:111965:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc48cc99c0 10000000:01000000:1.0:1788572579.490291:0:111965:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788572579.490299:0:111965:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc48cc99c0 from 1 10000000:01000000:1.0:1788572579.490302:0:111965:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788572579.490336:0:111965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.490347:0:111965:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.490358:0:111965:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572579.491685:0:111965:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6728b100 x1875449863943808/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572579.491698:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164300 going to sleep for 0 seconds 00000100:00100000:2.0:1788572579.506166:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572579.506187:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:2.0:1788572579.506196:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:2.0:1788572579.506202:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:2.0:1788572579.506207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0c000 MGS: changing import state from CONNECTING to FULL 10000000:01000000:2.0:1788572579.506210:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572579.506230:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:2.0:1788572579.506245:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000525ec5a2): req@ffff92fc6728b100 x1875449863943808/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572579.506262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c95aecce-2623-4667-b83f-7aa0f7bacb74:2392:1875449863943680:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:1.0:1788572579.506418:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.530907:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863943808:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572579.530942:0:111965:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.530957:0:111965:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572579.531037:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863943936:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572579.531140:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.540106:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863943936:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:1.0:1788572579.540153:0:111965:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788572579.540176:0:111965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.540188:0:111965:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572579.540197:0:111965:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc704fb000 10000000:01000000:1.0:1788572579.540203:0:111965:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:1.0:1788572579.540207:0:111965:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc704fb000 10000000:01000000:1.0:1788572579.540210:0:111965:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:1.0:1788572579.540214:0:111965:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc704fb000 10000000:01000000:1.0:1788572579.540216:0:111965:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:1.0:1788572579.540219:0:111965:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc704fb000 from 1 10000000:01000000:1.0:1788572579.540222:0:111965:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572579.540254:0:111965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.540262:0:111965:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.540273:0:111965:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572579.540294:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944064:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788572579.540364:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.547888:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944064:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572579.547918:0:111965:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.547931:0:111965:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572579.548010:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944192:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572579.548073:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.551345:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944192:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572579.551406:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944320:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572579.551500:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572579.562050:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944320:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572579.562408:0:111985:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111985:1875449863944448:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572579.562539:0:111985:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572579.565418:0:111985:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111985:1875449863944448:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572579.565465:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572579.565477:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572579.565480:0:111985:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:3.0:1788572579.565486:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc704fb000 00000020:00000080:3.0:1788572579.565489:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572579.565502:0:111985:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc704fb000 (ffff92fc700719c0) 00000020:00000080:3.0:1788572579.565523:0:111985:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:3.0:1788572579.565528:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc704fb000 00000020:00000080:3.0:1788572579.565531:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788572579.565879:0:111985:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:3.0:1788572579.565885:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572579.565889:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572579.565891:0:111985:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:3.0:1788572579.565894:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572579.565897:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572579.565899:0:111985:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:3.0:1788572579.565902:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc704fb000 00000020:00000080:3.0:1788572579.565904:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572579.565910:0:111985:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc704fb000 (ffff92fc70070000) 00000020:00000080:3.0:1788572579.565934:0:111985:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:3.0:1788572579.565940:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc704fb000 00000020:00000080:3.0:1788572579.565942:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:0.0:1788572579.577873:0:111985:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:0.0:1788572579.577889:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.577896:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.577898:0:111985:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:0.0:1788572579.577902:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572579.577906:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.577908:0:111985:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:0.0:1788572579.577912:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:0.0:1788572579.577914:0:111985:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:0.0:1788572579.577928:0:111985:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:0.0:1788572579.577935:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.577938:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.577940:0:111985:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:0.0:1788572579.577942:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572579.577946:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.577948:0:111985:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:0.0:1788572579.577950:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572579.577956:0:111985:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572579.577985:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.577989:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572579.578001:0:111985:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc704fb000 (ffff92fc70034d40) 00000020:00000080:0.0:1788572579.578020:0:111985:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:0.0:1788572579.578026:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.578028:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572579.578119:0:111985:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc48dc6800@lustre-MDT0000-mdc-ffff92fc704fb000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572579.579717:0:111985:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc704fb000-1 is initialized 00000040:01000000:0.0:1788572579.579981:0:111985:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc704fb000 ctxt 13 is initialized 00000020:00000080:0.0:1788572579.580161:0:111985:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:0.0:1788572579.580176:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc704fb000 00000020:00000080:0.0:1788572579.580180:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572579.580204:0:111985:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:0.0:1788572579.580236:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.580240:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.580242:0:111985:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:0.0:1788572579.580245:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572579.580248:0:111985:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:0.0:1788572579.580249:0:111985:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572579.580251:0:111985:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572579.580252:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572579.580255:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.580256:0:111985:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572579.580259:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572579.580264:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.580265:0:111985:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:0.0:1788572579.580269:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572579.580274:0:111985:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572579.580298:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.580301:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572579.580309:0:111985:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc704fb000 (ffff92fc700319c0) 00000020:00000080:0.0:1788572579.580320:0:111985:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:0.0:1788572579.580325:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.580328:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572579.580384:0:111985:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990a800@lustre-MDT0001-mdc-ffff92fc704fb000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572579.596847:0:111985:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc704fb000-2 is initialized 00000040:01000000:0.0:1788572579.597071:0:111985:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc704fb000 ctxt 13 is initialized 00000020:00000080:0.0:1788572579.597238:0:111985:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:0.0:1788572579.597256:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc704fb000 00000020:00000080:0.0:1788572579.597260:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:0.0:1788572579.597290:0:111985:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:0.0:1788572579.597329:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.597333:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.597335:0:111985:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:0.0:1788572579.597338:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572579.597342:0:111985:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:0.0:1788572579.597343:0:111985:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572579.597345:0:111985:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572579.597347:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572579.597348:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.597350:0:111985:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572579.597352:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572579.597357:0:111985:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre/d60.sanity-sec 00000020:00000080:0.0:1788572579.597361:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.597362:0:111985:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:0.0:1788572579.597365:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572579.597371:0:111985:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572579.597405:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.597409:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572579.597420:0:111985:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc704fb000 (ffff92fc70030000) 00000020:00000080:0.0:1788572579.597434:0:111985:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:0.0:1788572579.597440:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.597442:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572579.597504:0:111985:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990d800@lustre-OST0000-osc-ffff92fc704fb000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572579.597691:0:111985:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc704fb000-3 is initialized 00000020:00000080:0.0:1788572579.597919:0:111985:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:0.0:1788572579.597927:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc704fb000 00000020:00000080:0.0:1788572579.597929:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572579.597940:0:111985:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:0.0:1788572579.597945:0:111985:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc704fb000: Adding target lustre-OST0000_UUID 00020000:01000000:0.0:1788572579.597949:0:111985:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:0.0:1788572579.597959:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.597962:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.597963:0:111985:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:0.0:1788572579.597966:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:0.0:1788572579.597968:0:111985:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:0.0:1788572579.597970:0:111985:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572579.597971:0:111985:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:0.0:1788572579.597973:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:0.0:1788572579.597975:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.597976:0:111985:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:0.0:1788572579.597979:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:0.0:1788572579.597982:0:111985:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre/d60.sanity-sec 00000020:00000080:0.0:1788572579.597984:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.597985:0:111985:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:0.0:1788572579.597988:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:0.0:1788572579.597991:0:111985:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:0.0:1788572579.598003:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.598006:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:0.0:1788572579.598013:0:111985:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc704fb000 (ffff92fc70033380) 00000020:00000080:0.0:1788572579.598022:0:111985:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:0.0:1788572579.598027:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc704fb000 00000020:00000080:0.0:1788572579.598030:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:0.0:1788572579.598075:0:111985:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4990b000@lustre-OST0001-osc-ffff92fc704fb000: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572579.598282:0:111985:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc704fb000-4 is initialized 00000020:00000080:0.0:1788572579.598477:0:111985:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:0.0:1788572579.598485:0:111985:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc704fb000 00000020:00000080:0.0:1788572579.598488:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:0.0:1788572579.598496:0:111985:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:0.0:1788572579.598500:0:111985:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc704fb000: Adding target lustre-OST0001_UUID 00020000:01000000:0.0:1788572579.598503:0:111985:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:0.0:1788572579.598509:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.598513:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.598514:0:111985:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:0.0:1788572579.598518:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:0.0:1788572579.598520:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.598522:0:111985:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:0.0:1788572579.598525:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:0.0:1788572579.598527:0:111985:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:0.0:1788572579.598529:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:0.0:1788572579.598531:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:0.0:1788572579.598532:0:111985:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:0.0:1788572579.598582:0:111985:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111985:1875449863944576:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572579.606275:0:111985:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc458099c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.608877:0:111985:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111985:1875449863944576:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572579.608996:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572579.609009:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.609011:0:111985:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788572579.609018:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788572579.609061:0:111985:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111985:1875449863944704:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572579.609171:0:111985:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc511643c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.628502:0:111985:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111985:1875449863944704:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572579.628550:0:111985:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572579.628560:0:111985:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.628562:0:111985:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:1.0:1788572579.628567:0:111985:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:2.0:1788572579.630873:0:111965:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:2.0:1788572579.630902:0:111965:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572579.630936:0:111965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572579.630948:0:111965:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572579.630965:0:111965:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc704fb000 from 1 10000000:01000000:2.0:1788572579.630969:0:111965:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572579.631011:0:111965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572579.631019:0:111965:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572579.631029:0:111965:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572579.631060:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944832:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572579.631233:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859540 going to sleep for 11 seconds 00000100:00100000:3.0:1788572579.637510:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944832:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572579.637546:0:111965:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.637560:0:111965:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572579.638498:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944960:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788572579.638661:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.648352:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863944960:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:1.0:1788572579.649538:0:111965:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788572579.649633:0:111965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.649647:0:111965:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572579.649657:0:111965:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc704fb000 from 1 10000000:01000000:1.0:1788572579.649660:0:111965:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:1.0:1788572579.649699:0:111965:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.649713:0:111965:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.649758:0:111965:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572579.649784:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863945088:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:1.0:1788572579.649854:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.652275:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863945088:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:1.0:1788572579.652306:0:111965:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572579.652319:0:111965:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572579.652464:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863945216:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572579.654864:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.658625:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863945216:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:1.0:1788572579.658707:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863945344:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572579.658875:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc51164000 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.661253:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:c95aecce-2623-4667-b83f-7aa0f7bacb74:111965:1875449863945344:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788572579.662509:0:111992:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111992:1875449863945472:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572579.662772:0:111992:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc511643c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.668343:0:111992:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111992:1875449863945472:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572579.668452:0:111992:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572579.668460:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.668462:0:111992:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:1.0:1788572579.668468:0:111992:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc704fb000 00000020:00000080:1.0:1788572579.668470:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:1.0:1788572579.668476:0:111992:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572579.668477:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.668478:0:111992:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:1.0:1788572579.668480:0:111992:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572579.668481:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.668482:0:111992:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:1.0:1788572579.668484:0:111992:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc704fb000 00000020:00000080:1.0:1788572579.668485:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:1.0:1788572579.668566:0:111992:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572579.668570:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.668571:0:111992:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:1.0:1788572579.668574:0:111992:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572579.668576:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.668577:0:111992:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:1.0:1788572579.668580:0:111992:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc704fb000 00000020:00000080:1.0:1788572579.668583:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:1.0:1788572579.672140:0:111992:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 3548us 00000100:00100000:1.0:1788572579.672213:0:111992:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111992:1875449863945600:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572579.672403:0:111992:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc511643c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572579.674549:0:111992:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:c95aecce-2623-4667-b83f-7aa0f7bacb74:111992:1875449863945600:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572579.674589:0:111992:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572579.674596:0:111992:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572579.674598:0:111992:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:1.0:1788572579.674602:0:111992:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:3.0:1788572579.679822:0:111965:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:3.0:1788572579.679855:0:111965:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788572579.679884:0:111965:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572579.679898:0:111965:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:3.0:1788572579.679918:0:111965:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:0.0:1788572579.684838:0:111965:0:(genops.c:1385:class_connect()) connect: client 802259b2-6b09-4a33-8d4c-554d26b1ec81, cookie 0x58a604dcef352c74 00800000:01000000:0.0:1788572579.684867:0:111965:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 802259b2-6b09-4a33-8d4c-554d26b1ec81 to lustre-clilmv-ffff92fc704fb000 00800000:01000000:0.0:1788572579.684875:0:111965:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc704fb000(802259b2-6b09-4a33-8d4c-554d26b1ec81) - lustre-MDT0000_UUID, 802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:0.0:1788572579.684884:0:111965:0:(genops.c:1385:class_connect()) connect: client 802259b2-6b09-4a33-8d4c-554d26b1ec81, cookie 0x58a604dcef352c7b 00000100:00080000:0.0:1788572579.684896:0:111965:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc704fb000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572579.684908:0:111965:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc704fb000: import ffff92fc48dc6800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572579.684935:0:111965:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572579.685033:0:111965:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449863945728/t0(0) o38->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572579.685060:0:111965:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-MDT0000_UUID 00000100:00100000:3.0:1788572579.685171:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863945728:192.168.202.132@tcp:38:mount.lustre.0 00800000:01000000:0.0:1788572579.686138:0:111965:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc704fb000(802259b2-6b09-4a33-8d4c-554d26b1ec81) successfully (5) 00800000:00000004:0.0:1788572579.686143:0:111965:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc704fb000: age 3335, max_age 0 00000100:00080000:0.0:1788572579.686199:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67049f80 x1875449863945856/t0(0) o41->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00800000:01000000:0.0:1788572579.686238:0:111965:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc704fb000(802259b2-6b09-4a33-8d4c-554d26b1ec81) - lustre-MDT0001_UUID, 802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:0.0:1788572579.686248:0:111965:0:(genops.c:1385:class_connect()) connect: client 802259b2-6b09-4a33-8d4c-554d26b1ec81, cookie 0x58a604dcef352c82 00000100:00080000:0.0:1788572579.686254:0:111965:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc704fb000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:0.0:1788572579.686261:0:111965:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc704fb000: import ffff92fc4990a800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572579.686270:0:111965:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788572579.686283:0:111965:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863945984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572579.686301:0:111965:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-MDT0001_UUID 00000100:00100000:3.0:1788572579.686356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863945984:192.168.202.132@tcp:38:mount.lustre.0 00800000:01000000:0.0:1788572579.688043:0:111965:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc704fb000(802259b2-6b09-4a33-8d4c-554d26b1ec81) successfully (5) 00800000:00000004:0.0:1788572579.688049:0:111965:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc704fb000: age 3335, max_age 0 00000100:00080000:0.0:1788572579.688115:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704aa00 x1875449863946112/t0(0) o41->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000080:00000004:0.0:1788572579.688158:0:111965:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc704fb000: age 3335, max_age 4334 00800000:00000004:0.0:1788572579.688164:0:111965:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc704fb000: age 3335, max_age 4334 00000100:00080000:0.0:1788572579.688184:0:111965:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704a300 x1875449863946240/t0(0) o41->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572579.688193:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809480 going to sleep for 0 seconds 00000100:00100000:3.0:1788572579.710076:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc704fb000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572579.710101:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc704fb000: connect to target with instance 1 00000100:00080000:3.0:1788572579.710111:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc704fb000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:0.0:1788572579.714840:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:0.0:1788572579.714851:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc6800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:0.0:1788572579.716050:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f7e70609): req@ffff92fc67049f80 x1875449863945856/t0(0) o41->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572579.716073:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000ebc23ee6): req@ffff92fc6704a300 x1875449863946240/t0(0) o41->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572579.716086:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863945728:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788572579.716342:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572579.716393:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc704fb000: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572579.716404:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc704fb000: connect to target with instance 1 00000100:00080000:0.0:1788572579.716413:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc704fb000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:0.0:1788572579.716422:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:0.0:1788572579.716425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00080000:0.0:1788572579.718472:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002939d3c5): req@ffff92fc6704aa00 x1875449863946112/t0(0) o41->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572579.718495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863945984:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572579.730900:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:0:1875449863945856:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572579.730966:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:0:1875449863946112:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572579.733937:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946240:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:3.0:1788572579.734104:0:111965:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc704fb000: update ffff92fc70035198 cache blocks 317009/353529 objects 1023700/1024000 00000080:00000004:3.0:1788572579.734112:0:111965:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc704fb000: update ffff92fc70070458 cache blocks 317009/353529 objects 1023700/1024000 00000100:00100000:3.0:1788572579.734162:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946368:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:3.0:1788572579.734245:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572579.737146:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946368:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:3.0:1788572579.737175:0:111965:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000422:0x1:0x0] 00000080:00100000:3.0:1788572579.737189:0:111965:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:3.0:1788572579.737193:0:111965:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:3.0:1788572579.737205:0:111965:0:(genops.c:1385:class_connect()) connect: client 802259b2-6b09-4a33-8d4c-554d26b1ec81, cookie 0x58a604dcef352c89 00000020:00000080:3.0:1788572579.737222:0:111965:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352c90 00000100:00080000:3.0:1788572579.737229:0:111965:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc704fb000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572579.737238:0:111965:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc704fb000: import ffff92fc4990d800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572579.737251:0:111965:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572579.737279:0:111965:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449863946496/t0(0) o8->lustre-OST0000-osc-ffff92fc704fb000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572579.737308:0:111965:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-OST0000_UUID 00020000:01000000:3.0:1788572579.737330:0:111965:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc704fb000) active 00000020:00000080:3.0:1788572579.737357:0:111965:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352c97 00000100:00080000:3.0:1788572579.737363:0:111965:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc704fb000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572579.737369:0:111965:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc704fb000: import ffff92fc4990b000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572579.737374:0:111965:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572579.737388:0:111965:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449863946624/t0(0) o8->lustre-OST0001-osc-ffff92fc704fb000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572579.737397:0:111965:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-OST0001_UUID 00020000:01000000:3.0:1788572579.737402:0:111965:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc704fb000) active 00000100:00100000:3.0:1788572579.737441:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946752:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788572579.737503:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572579.739869:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863946496:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:2.0:1788572579.740209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863946624:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572579.743371:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946752:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788572579.743450:0:111965:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946880:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:3.0:1788572579.743614:0:111965:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572579.749476:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc704fb000: committing for last_committed 0 gen 1 00000100:00080000:2.0:1788572579.749498:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc704fb000: connect to target with instance 1 00000100:00080000:2.0:1788572579.749511:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc704fb000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572579.749525:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572579.749529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990d800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572579.749538:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1788572579.749542:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572579.749548:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572579.749565:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863946496:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:1.0:1788572579.754909:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc704fb000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572579.754928:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc704fb000: connect to target with instance 1 00000100:00080000:1.0:1788572579.754938:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc704fb000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572579.754950:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572579.754964:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990b000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:1.0:1788572579.754971:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:1.0:1788572579.754976:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572579.754981:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:1.0:1788572579.754995:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863946624:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572579.762502:0:111965:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:802259b2-6b09-4a33-8d4c-554d26b1ec81:111965:1875449863946880:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:3.0:1788572579.762679:0:111965:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00200000:3.0:1788572579.762687:0:111965:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000080:02000400:3.0:1788572579.762829:0:111965:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:3.0:1788572579.762834:0:111965:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre/d60.sanity-sec: Mount complete 00000080:00200008:1.0:1788572580.054228:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), request_mask 606, flags 0, foreign 0 00000080:00200000:1.0:1788572580.054243:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:1.0:1788572580.054268:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572580.054327:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.054339:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.054348:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572580.054375:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947008:192.168.202.132@tcp:101:ls.0 00000100:00100000:1.0:1788572580.054461:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ae40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.058889:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947008:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.058918:0:112010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.058931:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.059060:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.059090:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.059104:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.059112:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.059120:0:112010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572580.059133:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67288a80 x1875449863947008/t0(0) o101->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572580.059146:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.059159:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc460d7000 type 13 mode 4 res 8589935650/1 (0 0) 00000080:00200000:2.0:1788572580.059165:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:2.0:1788572580.059220:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947136:192.168.202.132@tcp:900:ls.0 00000100:00100000:2.0:1788572580.059287:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.061230:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947136:192.168.202.132@tcp:900:ls.0 00000100:00100000:2.0:1788572580.061291:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947264:192.168.202.132@tcp:33:ls.0 00000100:00100000:2.0:1788572580.061351:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.067717:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947264:192.168.202.132@tcp:33:ls.0 00000080:00200000:2.0:1788572580.067810:0:112010:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572580.067819:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:2.0:1788572580.067827:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:2.0:1788572580.067830:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bcb8 00010000:00010000:2.0:1788572580.067836:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.067849:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:2.0:1788572580.068091:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572580.068142:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.068151:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.068161:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572580.068181:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947392:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572580.068257:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.080408:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947392:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.080439:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.080455:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.080467:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572580.080481:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc4617d500 x1875449863947392/t0(0) o101->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 568/1040 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572580.080493:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.080507:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc460d7000 type 13 mode 4 res 8589935650/1 (0 0) 00000080:00200000:2.0:1788572580.080513:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572580.080536:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352ca5 00000080:00010000:2.0:1788572580.080543:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ca5 from it ffff9e038231bb80 00010000:00010000:2.0:1788572580.080547:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.080557:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572580.145298:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:1.0:1788572580.145329:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572580.145365:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572580.145382:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572580.145390:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572580.145398:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:1.0:1788572580.145401:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:1.0:1788572580.145409:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bbb0 00010000:00010000:1.0:1788572580.145414:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572580.145423:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572580.145442:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572580.145453:0:112010:0:(file.c:1090:ll_file_open()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), flags 304000 00000080:00200008:1.0:1788572580.145467:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), request_mask 2047, flags 0, foreign 0 00000002:00010000:1.0:1788572580.145485:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572580.145493:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572580.145501:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000100:00100000:1.0:1788572580.146156:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947520:192.168.202.132@tcp:37:ls.0 00000100:00100000:1.0:1788572580.146247:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9a3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572580.170374:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947520:192.168.202.132@tcp:37:ls.0 00010000:00010000:3.0:1788572580.171316:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572580.171334:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572580.171381:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:3.0:1788572580.171403:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572580.171420:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572580.171431:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572580.171439:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572580.171445:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572580.171449:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:3.0:1788572580.171455:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bcd8 00010000:00010000:3.0:1788572580.171458:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572580.171466:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200008:3.0:1788572580.171555:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), request_mask 256, flags 0, foreign 0 00000080:00200000:3.0:1788572580.171560:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:3.0:1788572580.171633:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572580.171642:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572580.171650:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572580.171657:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572580.171661:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572580.171664:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:3.0:1788572580.171669:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bcb8 00010000:00010000:3.0:1788572580.171672:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572580.171680:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572580.171927:0:112010:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88) pos/size0/4096 32bit_api 0 00000080:00200000:3.0:1788572580.172000:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:3.0:1788572580.172008:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572580.172016:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572580.172025:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572580.172031:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572580.172034:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572580.172037:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:3.0:1788572580.172041:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bb58 00010000:00010000:3.0:1788572580.172044:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572580.172051:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572580.172060:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572580.172087:0:112010:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=MmyUdhnkoTbIcOsepdJyuA, dir=[0x200000422:0x1:0x0](ffff92fc665dba88), flags=8 00000080:00200000:3.0:1788572580.172091:0:112010:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=MmyUdhnkoTbIcOsepdJyuA, dir=[0x200000422:0x1:0x0](ffff92fc665dba88), intent=getattr 00000002:00010000:3.0:1788572580.172115:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 2l”vä¡6Èpë¥Òr¸,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572580.172157:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.172166:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.172175:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572580.172267:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947648:192.168.202.132@tcp:101:ls.0 00000100:00100000:3.0:1788572580.172416:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572580.177324:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947648:192.168.202.132@tcp:101:ls.0 00010000:00010000:3.0:1788572580.177366:0:112010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177391:0:112010:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177400:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177412:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177424:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177438:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177446:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177454:0:112010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572580.177467:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733aa00 x1875449863947648/t0(0) o101->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/672 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572580.177478:0:112010:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572580.177491:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177503:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc460d7000 type 13 mode 4 res 8589935650/2 (0 0) 00000080:00200000:3.0:1788572580.177510:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572580.177541:0:112010:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dd348) 00000080:00200000:3.0:1788572580.177802:0:112010:0:(file.c:6843:ll_layout_conf()) [0x200000422:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788572580.177816:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dd348) for lock 0x58a604dcef352cac 00000080:00010000:3.0:1788572580.177826:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572580.177831:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cac from it ffff9e038231bb20 00010000:00010000:3.0:1788572580.177834:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.177848:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572580.177888:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file MmyUdhnkoTbIcOsepdJyuA:[0x200000422:0x2:0x0](ffff92fc665dd348), request_mask 606, flags 0, foreign 0 00000080:00200000:3.0:1788572580.177895:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dd348),name=MmyUdhnkoTbIcOsepdJyuA 00000002:00010000:3.0:1788572580.178424:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x2:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572580.178438:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.178450:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572580.178457:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572580.178462:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572580.178465:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dd348) for lock 0x58a604dcef352cac 00000080:00010000:3.0:1788572580.178470:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cac from it ffff9e038231bcb8 00010000:00010000:3.0:1788572580.178473:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.178481:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572580.178500:0:112010:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788572580.178510:0:112010:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000422:0x2:0x0] 00000008:00010000:3.0:1788572580.178545:0:112010:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572580.178548:0:112010:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc7049dab0, osc lock ffff92fc4645cd10, flags 0x400001002 00010000:00010000:3.0:1788572580.178674:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.178685:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572580.178798:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863947776:192.168.202.132@tcp:101:ls.0 00000020:00200000:3.0:1788572580.178934:0:112010:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788572580.185972:0:2395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.186001:0:2395:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.186023:0:2395:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.186039:0:2395:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572580.186053:0:2395:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2, setting rss=2; kms=2, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572580.186064:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572580.186089:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.186097:0:2395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.186104:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.186113:0:2395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572580.186133:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863947776:192.168.202.132@tcp:101:ls.0 00000080:00200000:3.0:1788572580.186282:0:112010:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000422:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:3.0:1788572580.186290:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc67174de0 (ffff92fc6713b518/0) at cl_lock_release():245 00000020:00010000:3.0:1788572580.186293:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67174de0 (ffff92fc6713b518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572580.186299:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049dab0 (ffff92fc6713b368/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572580.186304:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67174de0 (ffff92fc6713b518/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572580.186307:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049dab0 (ffff92fc6713b518/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572580.186323:0:112010:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572580.186385:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:3.0:1788572580.186411:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572580.186428:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572580.186442:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572580.186450:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572580.186456:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572580.186461:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:3.0:1788572580.186467:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bbf8 00010000:00010000:3.0:1788572580.186471:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572580.186483:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572580.186494:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572580.186501:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=MmyUdhnkoTbIcOsepdJyuA, flags=9 00000080:00200000:3.0:1788572580.186576:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:3.0:1788572580.186585:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572580.186592:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572580.186600:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572580.186607:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572580.186611:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572580.186614:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:3.0:1788572580.186618:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bb58 00010000:00010000:3.0:1788572580.186620:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572580.186628:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572580.186636:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572580.186651:0:112010:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, dir=[0x200000422:0x1:0x0](ffff92fc665dba88), flags=8 00000080:00200000:3.0:1788572580.186656:0:112010:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, dir=[0x200000422:0x1:0x0](ffff92fc665dba88), intent=getattr 00000002:00010000:3.0:1788572580.186682:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: Ì™C¸ æºî×Wè9f,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572580.186942:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.186952:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572580.186964:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572580.187398:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947904:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572580.187543:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.196640:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863947904:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.196671:0:112010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196691:0:112010:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196700:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196711:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196745:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196758:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196765:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196773:0:112010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572580.196790:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6733b800 x1875449863947904/t0(0) o101->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572580.196802:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.196870:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc460d7000 type 13 mode 4 res 9663677451/1 (0 0) 00000100:00100000:2.0:1788572580.196921:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948032:192.168.202.132@tcp:900:ls.0 00000100:00100000:2.0:1788572580.197139:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.204127:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948032:192.168.202.132@tcp:900:ls.0 00000002:00010000:2.0:1788572580.204169:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572580.204218:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.204228:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.204237:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572580.204255:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948160:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572580.204321:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.213287:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948160:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.213317:0:112010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 4/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213335:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213358:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213374:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213387:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213395:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213403:0:112010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572580.213414:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617d500 x1875449863948160/t0(0) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572580.213426:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213439:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc45857400 type 13 mode 4 res 9663677451/1 (0 0) 00000080:00200000:2.0:1788572580.213459:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572580.213504:0:112010:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88) 00000080:00200000:2.0:1788572580.213513:0:112010:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040b:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572580.213531:0:112010:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) for remote lock 0x58a604dcef352cc1 00000080:00010000:2.0:1788572580.213536:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef352cba 00000080:00010000:2.0:1788572580.213547:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572580.213550:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cba from it ffff9e038231bb20 00010000:00010000:2.0:1788572580.213554:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213563:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572580.213588:0:112010:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x58a604dcef352cc1 from it ffff9e038231bb20 00010000:00010000:2.0:1788572580.213592:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.213599:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572580.213625:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file zJlDuKDmuu6NAddX6DlmGw:[0x24000040b:0x1:0x0](ffff92fc665e5b88), request_mask 606, flags 0, foreign 0 00000080:00200000:2.0:1788572580.213632:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88),name=zJlDuKDmuu6NAddX6DlmGw 00000002:00010000:2.0:1788572580.213690:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572580.213700:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572580.213709:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572580.213716:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572580.213755:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572580.213759:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef352cc1 00000080:00010000:2.0:1788572580.213764:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cc1 from it ffff9e038231bcb8 00010000:00010000:2.0:1788572580.213767:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572580.213774:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572580.213814:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:2.0:1788572580.213822:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572580.213830:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572580.213839:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572580.213845:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572580.213849:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:2.0:1788572580.213851:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:2.0:1788572580.213854:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bbf8 00010000:00010000:2.0:1788572580.213857:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572580.213864:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572580.213872:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572580.213878:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, flags=9 00000080:00200000:2.0:1788572580.213903:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:2.0:1788572580.213909:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572580.213915:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572580.213924:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572580.213930:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572580.213946:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:2.0:1788572580.213949:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:2.0:1788572580.213953:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bbf8 00010000:00010000:2.0:1788572580.213956:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572580.213964:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572580.213972:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572580.213976:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, flags=9 00000002:00010000:2.0:1788572580.213998:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572580.214035:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.214043:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.214052:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572580.214084:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948288:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572580.214158:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.228206:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948288:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.228239:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.228256:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.228268:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572580.228282:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc4617d500 x1875449863948288/t0(0) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 568/1048 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572580.228294:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.228307:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc45857400 type 13 mode 4 res 9663677451/1 (0 0) 00000080:00200000:2.0:1788572580.228313:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x24000040b:0x1:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572580.228336:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef352cc8 00000080:00010000:2.0:1788572580.228345:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cc8 from it ffff9e038231bb80 00010000:00010000:2.0:1788572580.228349:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.228359:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572580.228418:0:112010:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:2.0:1788572580.228506:0:112010:0:(file.c:479:ll_file_release()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), flags 304000 00000080:00200000:0.0:1788572580.251983:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:0.0:1788572580.252023:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572580.252048:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788572580.252066:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788572580.252145:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572580.252153:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:0.0:1788572580.252157:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:0.0:1788572580.252164:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bbf8 00010000:00010000:0.0:1788572580.252169:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788572580.252179:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788572580.252196:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572580.252205:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, flags=267 00000080:00200000:0.0:1788572580.252240:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 44 00000080:00200008:0.0:1788572580.252247:0:112010:0:(file.c:1090:ll_file_open()) START file zJlDuKDmuu6NAddX6DlmGw:[0x24000040b:0x1:0x0](ffff92fc665e5b88), flags 304000 00000002:00010000:0.0:1788572580.252264:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788572580.252323:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572580.252331:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572580.252342:0:112010:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc704fb000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572580.252355:0:112010:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704a680 x1875449863948544/t0(0) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/131800 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572580.252364:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572580.252394:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948544:192.168.202.132@tcp:101:ls.0 00000100:00100000:0.0:1788572580.252668:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572580.260462:0:112010:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc704fb000: committing for last_committed 4294976775 gen 1 00000100:00100000:1.0:1788572580.260481:0:112010:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc6704a680 x1875449863948544/t4294976775(4294976775) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572596 ref 3 fl Rpc:RPQU/604/0 rc 0/301 job:'ls.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572580.260494:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948544:192.168.202.132@tcp:101:ls.0 00010000:00010000:1.0:1788572580.260512:0:112010:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260526:0:112010:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260534:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260542:0:112010:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 3/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260552:0:112010:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 4/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260559:0:112010:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc2e00 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:1.0:1788572580.260621:0:112010:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 4/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260629:0:112010:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 4/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260636:0:112010:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 5/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260643:0:112010:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572580.260650:0:112010:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 3/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.260657:0:112010:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc51bc2e00/0x58a604dcef352ccf lrc: 0/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572580.260687:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc6704a680 x1875449863948544/t4294976775(4294976775) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572596 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'ls.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572580.260700:0:112010:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6704a680 x1875449863948544/t4294976775(4294976775) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572596 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'ls.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572580.260713:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572580.260965:0:112010:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040b:0x1:0x0], updating i_size 4096 00000002:00100000:1.0:1788572580.260987:0:112010:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc6704a680 x1875449863948544/t4294976775(4294976775) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572596 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'ls.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572580.261433:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file zJlDuKDmuu6NAddX6DlmGw:[0x24000040b:0x1:0x0](ffff92fc665e5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572580.261442:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88),name=zJlDuKDmuu6NAddX6DlmGw 00000002:00010000:1.0:1788572580.261498:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572580.261512:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572580.261525:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572580.261533:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572580.261539:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:1.0:1788572580.261543:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef352cc1 00000080:00010000:1.0:1788572580.261549:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cc1 from it ffff9e038231bcd8 00010000:00010000:1.0:1788572580.261553:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572580.261562:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200008:1.0:1788572580.261911:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file zJlDuKDmuu6NAddX6DlmGw:[0x24000040b:0x1:0x0](ffff92fc665e5b88), request_mask 256, flags 0, foreign 0 00000080:00200000:2.0:1788572580.261994:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88),name=zJlDuKDmuu6NAddX6DlmGw 00000002:00010000:2.0:1788572580.262025:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572580.262034:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572580.262043:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572580.262050:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572580.262054:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) 00000080:00010000:2.0:1788572580.262057:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e5b88) for lock 0x58a604dcef352cc1 00000080:00010000:2.0:1788572580.262062:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cc1 from it ffff9e038231bcb8 00010000:00010000:2.0:1788572580.262065:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572580.262083:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572580.262119:0:112010:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788572580.262143:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x24000040b:0x1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572580.262151:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572580.262159:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000100:00100000:2.0:1788572580.263156:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948672:192.168.202.132@tcp:37:ls.0 00000100:00100000:2.0:1788572580.263279:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.273266:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948672:192.168.202.132@tcp:37:ls.0 00010000:00010000:2.0:1788572580.273959:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572580.273977:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572580.274331:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:2.0:1788572580.274351:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572580.274366:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572580.274379:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572580.274386:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572580.274391:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:2.0:1788572580.274395:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:2.0:1788572580.274401:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bb58 00010000:00010000:2.0:1788572580.274404:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788572580.274412:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572580.274423:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572580.274429:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, flags=24 00000080:00200000:2.0:1788572580.274456:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572580.274469:0:112010:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=RKjmNSpYZFE-XEs3hBF_Eg, dir=[0x24000040b:0x1:0x0](ffff92fc665e5b88), flags=8 00000080:00200000:2.0:1788572580.274473:0:112010:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=RKjmNSpYZFE-XEs3hBF_Eg, dir=[0x24000040b:0x1:0x0](ffff92fc665e5b88), intent=getattr 00000002:00010000:2.0:1788572580.274494:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: D¨æ5*XdQ>\K7„=¿,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572580.274533:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.274541:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.274550:0:112010:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112010 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572580.274572:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948800:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572580.274702:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49859480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572580.281628:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863948800:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.281659:0:112010:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 4/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281685:0:112010:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281693:0:112010:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281712:0:112010:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281763:0:112010:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281778:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281786:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281794:0:112010:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572580.281805:0:112010:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617d500 x1875449863948800/t0(0) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/672 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572580.281817:0:112010:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572580.281831:0:112010:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.281843:0:112010:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc45857400 type 13 mode 4 res 9663677451/2 (0 0) 00000080:00200000:2.0:1788572580.281850:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x24000040b:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572580.281890:0:112010:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665e6c08) 00000080:00200000:2.0:1788572580.281995:0:112010:0:(file.c:6843:ll_layout_conf()) [0x24000040b:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572580.282020:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352cd6 00000080:00010000:2.0:1788572580.282033:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665e6c08) 00000080:00010000:2.0:1788572580.282038:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cd6 from it ffff9e038231bb20 00010000:00010000:2.0:1788572580.282042:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.282052:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572580.282300:0:112010:0:(file.c:6296:ll_getattr_dentry()) START file RKjmNSpYZFE-XEs3hBF_Eg:[0x24000040b:0x2:0x0](ffff92fc665e6c08), request_mask 606, flags 0, foreign 0 00000080:00200000:2.0:1788572580.283393:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665e6c08),name=RKjmNSpYZFE-XEs3hBF_Eg 00000002:00010000:2.0:1788572580.283454:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x2:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572580.283470:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572580.283485:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572580.283494:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x24000040b:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572580.283500:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665e6c08) 00000080:00010000:2.0:1788572580.283503:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665e6c08) for lock 0x58a604dcef352cd6 00000080:00010000:2.0:1788572580.283510:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352cd6 from it ffff9e038231bcb8 00010000:00010000:2.0:1788572580.283514:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572580.283524:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572580.283553:0:112010:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788572580.283563:0:112010:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040b:0x2:0x0] 00000008:00010000:2.0:1788572580.283603:0:112010:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788572580.283607:0:112010:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc705504b0, osc lock ffff92fc583717c0, flags 0x400001002 00010000:00010000:2.0:1788572580.283656:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.283667:0:112010:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 3/1,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572580.283706:0:112010:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788572580.283718:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572580.283771:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863948928:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572580.285952:0:2395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.285982:0:2395:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.285997:0:2395:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.286080:0:2395:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788572580.286099:0:2395:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2, setting rss=2; kms=2, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788572580.286110:0:2395:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788572580.286129:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.286138:0:2395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.286146:0:2395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.286154:0:2395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788572580.286173:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863948928:192.168.202.132@tcp:101:ls.0 00000080:00200000:0.0:1788572580.286253:0:112010:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040b:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:0.0:1788572580.286261:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc70474000 (ffff92fc6721a128/0) at cl_lock_release():245 00000020:00010000:0.0:1788572580.286265:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc70474000 (ffff92fc6721a128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572580.286270:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc705504b0 (ffff92fc6721a638/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572580.286275:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc70474000 (ffff92fc6721a128/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572580.286278:0:112010:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc705504b0 (ffff92fc6721a128/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572580.286292:0:112010:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572580.286342:0:112010:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:0.0:1788572580.286363:0:112010:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572580.286379:0:112010:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572580.286392:0:112010:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572580.286400:0:112010:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572580.286409:0:112010:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:0.0:1788572580.286413:0:112010:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352c9e 00000080:00010000:0.0:1788572580.286418:0:112010:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352c9e from it ffff9e038231bbf8 00010000:00010000:0.0:1788572580.286422:0:112010:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788572580.286431:0:112010:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788572580.286448:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572580.286454:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=zJlDuKDmuu6NAddX6DlmGw, flags=25 00000080:00200000:0.0:1788572580.286482:0:112010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88), inode mode 41ed mask 1 00000080:00200000:0.0:1788572580.286486:0:112010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=RKjmNSpYZFE-XEs3hBF_Eg, flags=9 00000080:00200000:0.0:1788572580.286538:0:112010:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:0.0:1788572580.286565:0:112010:0:(file.c:479:ll_file_release()) START file zJlDuKDmuu6NAddX6DlmGw:[0x24000040b:0x1:0x0](ffff92fc665e5b88), flags 304000 00000002:00100000:0.0:1788572580.286617:0:112010:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc6704a680 x1875449863948544/t4294976775(4294976775) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572596 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'ls.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572580.286629:0:112010:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc704fb000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572580.286635:0:112010:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67048a80 x1875449863949184/t0(0) o35->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572580.286657:0:112010:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863949184:192.168.202.132@tcp:35:ls.0 00000100:00100000:0.0:1788572580.286757:0:112010:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572580.290385:0:112010:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ls:802259b2-6b09-4a33-8d4c-554d26b1ec81:112010:1875449863949184:192.168.202.132@tcp:35:ls.0 00000002:00100000:0.0:1788572580.290420:0:112010:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc6704a680 x1875449863948544/t4294976775(4294976775) o101->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572596 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'ls.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572580.302219:0:112014:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace MGC192.168.202.132@tcp 00010000:00010000:1.0:1788572580.302257:0:112014:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff92fc704fb000 00010000:00010000:1.0:1788572580.302286:0:112014:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/4 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572580.302572:0:112016:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0001-mdc-ffff92fc704fb000 00010000:00010000:1.0:1788572580.302591:0:112016:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/7 locks to regular blp list, 2 blwis in pool 00010000:00010000:3.0:1788572580.302654:0:111978:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 4 locks of 3 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788572580.302665:0:111978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572580.302772:0:111978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.302788:0:111978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572580.302803:0:111978:0:(vvp_object.c:117:vvp_conf_set()) [0x200000422:0x2:0x0]: losing layout lock 00010000:00010000:3.0:1788572580.302835:0:111978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.302884:0:111978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 1/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572580.302893:0:111978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.302900:0:111978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 1/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572580.302909:0:111978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 1/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.302915:0:111978:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00000100:00100000:3.0:1788572580.302936:0:111978:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:802259b2-6b09-4a33-8d4c-554d26b1ec81:111978:1875449863949312:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572580.303010:0:111978:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053a80 going to sleep for 11 seconds 00010000:00010000:1.0:1788572580.303981:0:111977:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572580.305360:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.305372:0:111977:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572580.305412:0:111977:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572580.305451:0:111977:0:(vvp_object.c:117:vvp_conf_set()) [0x24000040b:0x2:0x0]: losing layout lock 00010000:00010000:1.0:1788572580.305498:0:111977:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572580.305542:0:111977:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572580.305551:0:111977:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572580.305571:0:111977:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572580.305577:0:111977:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00000100:00100000:1.0:1788572580.305597:0:111977:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:111977:1875449863949440:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572580.305660:0:111977:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53e9ae40 going to sleep for 11 seconds 00010000:00010000:1.0:1788572580.305775:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.305778:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572580.306169:0:112015:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0000-osc-ffff92fc704fb000 00010000:00010000:2.0:1788572580.306194:0:112015:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788572580.306250:0:112018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00000100:00100000:3.0:1788572580.308057:0:111978:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_02:802259b2-6b09-4a33-8d4c-554d26b1ec81:111978:1875449863949312:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572580.308096:0:111978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc702f5e00/0x58a604dcef352c9e lrc: 0/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793221 expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572580.308112:0:111978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204e00/0x58a604dcef352cba lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179326e expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.308130:0:111978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70205c00/0x58a604dcef352cac lrc: 0/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a22179324b expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572580.308143:0:111978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70366200/0x58a604dcef352ca5 lrc: 0/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179322f expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.308165:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572580.308167:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572580.308170:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.308618:0:112014:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-OST0001-osc-ffff92fc704fb000 00010000:00010000:1.0:1788572580.309435:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.309439:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.309441:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572580.309518:0:111977:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:111977:1875449863949440:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572580.309542:0:111977:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70366e00/0x58a604dcef352cc1 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179327c expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572580.309555:0:111977:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367200/0x58a604dcef352cd6 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xc9a239a2217932b4 expref: -99 pid: 112010 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572580.309588:0:111977:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc704fb000 lock: ffff92fc70367600/0x58a604dcef352cc8 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179328a expref: -99 pid: 112010 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572580.309609:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572580.309610:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572580.309612:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572580.309913:0:112018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400010000 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572580.309938:0:112018:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572580.309951:0:112018:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1788572580.309979:0:112018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400010000 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572580.309989:0:112018:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572580.309993:0:112018:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1788572580.310038:0:112018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 1/0,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.310047:0:112018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.310054:0:112018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00000100:00100000:3.0:1788572580.310081:0:112018:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:802259b2-6b09-4a33-8d4c-554d26b1ec81:112018:1875449863949568:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572580.310160:0:112018:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572580.312576:0:112018:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:802259b2-6b09-4a33-8d4c-554d26b1ec81:112018:1875449863949568:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572580.312606:0:112018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000049e7d039/0x58a604dcef352cdd lrc: 0/0,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a2217932bb expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.312633:0:112018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff92fc704fb000 lock: 0000000004e2688b/0x58a604dcef352cb3 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400010020 nid: local remote: 0xc9a239a221793252 expref: -99 pid: 112010 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572580.312659:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572580.312661:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572580.312664:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572580.423443:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e5b88) 00000080:00200000:3.0:1788572580.423485:0:111676:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572580.423497:0:111676:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572580.423524:0:111676:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572580.423529:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665e6c08) 00000080:00200000:3.0:1788572580.423648:0:111676:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572580.423653:0:111676:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572580.423662:0:111676:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572580.423666:0:111676:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dd348) 00010000:00010000:3.0:1788572580.587286:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572581.572825:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572581.784195:0:112041:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572581.784211:0:112041:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:3.0:1788572581.784240:0:112041:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572581.784307:0:112041:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112041 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.784317:0:112041:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112041 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.784328:0:112041:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112041 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572581.784355:0:112041:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job lsof:802259b2-6b09-4a33-8d4c-554d26b1ec81:112041:1875449863949696:192.168.202.132@tcp:101:lsof.0 00000100:00100000:3.0:1788572581.784470:0:112041:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572581.793844:0:112041:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job lsof:802259b2-6b09-4a33-8d4c-554d26b1ec81:112041:1875449863949696:192.168.202.132@tcp:101:lsof.0 00010000:00010000:3.0:1788572581.793995:0:112041:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794079:0:112041:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794094:0:112041:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794105:0:112041:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794120:0:112041:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794127:0:112041:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794134:0:112041:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572581.794146:0:112041:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67338700 x1875449863949696/t0(0) o101->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572581.794157:0:112041:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.794169:0:112041:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc460d7000 type 13 mode 4 res 8589935650/1 (0 0) 00000080:00200000:3.0:1788572581.794175:0:112041:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:3.0:1788572581.794245:0:112041:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job lsof:802259b2-6b09-4a33-8d4c-554d26b1ec81:112041:1875449863949824:192.168.202.132@tcp:33:lsof.0 00000100:00100000:3.0:1788572581.794378:0:112041:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc48053e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572581.806358:0:112041:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job lsof:802259b2-6b09-4a33-8d4c-554d26b1ec81:112041:1875449863949824:192.168.202.132@tcp:33:lsof.0 00000080:00200000:3.0:1788572581.806417:0:112041:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572581.806429:0:112041:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352ce4 00000080:00010000:3.0:1788572581.806437:0:112041:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572581.806441:0:112041:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ce4 from it ffff9e0382f3bcb8 00010000:00010000:3.0:1788572581.806448:0:112041:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572581.806463:0:112041:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572582.596955:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572583.621148:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572584.645881:0:111978:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572584.969930:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:c95aecce-2623-4667-b83f-7aa0f7bacb74:2395:1875449863949952:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572584.974963:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:2396:1875449863950080:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.975289:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863950208:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.975323:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863950464:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.975779:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:c95aecce-2623-4667-b83f-7aa0f7bacb74:2395:1875449863949952:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572584.985248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:2396:1875449863950336:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.985948:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863950208:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.985962:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:802259b2-6b09-4a33-8d4c-554d26b1ec81:2395:1875449863950464:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572584.986460:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:2396:1875449863950080:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.991413:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:802259b2-6b09-4a33-8d4c-554d26b1ec81:2396:1875449863950336:192.168.202.132@tcp:400:kworker.0 00000080:00200008:2.0:1788572585.025479:0:112049:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572585.025491:0:112049:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:2.0:1788572585.025513:0:112049:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572585.025532:0:112049:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.025547:0:112049:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572585.025555:0:112049:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572585.025563:0:112049:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:2.0:1788572585.025566:0:112049:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352ce4 00000080:00010000:2.0:1788572585.025573:0:112049:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ce4 from it ffff9e0382f53cb8 00010000:00010000:2.0:1788572585.025577:0:112049:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572585.025586:0:112049:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788572585.025755:0:112049:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572585.025760:0:112049:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88),name=/ 00000002:00010000:2.0:1788572585.025769:0:112049:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572585.025777:0:112049:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572585.025785:0:112049:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572585.025791:0:112049:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572585.025795:0:112049:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00010000:2.0:1788572585.025797:0:112049:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dba88) for lock 0x58a604dcef352ce4 00000080:00010000:2.0:1788572585.025801:0:112049:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352ce4 from it ffff9e0382f53cb8 00010000:00010000:2.0:1788572585.025803:0:112049:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572585.025816:0:112049:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572585.039816:0:112049:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dba88) 00000080:00200000:0.0:1788572585.039857:0:112049:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff92fc704fb000 (sb ffff92fc704fb000) 00010000:00010000:0.0:1788572585.039908:0:112049:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039919:0:112049:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039928:0:112049:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039935:0:112049:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039955:0:112049:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788572585.039960:0:112049:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039967:0:112049:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039973:0:112049:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039979:0:112049:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.039990:0:112049:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 00010000:00010000:0.0:1788572585.039995:0:112049:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040001:0:112049:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040007:0:112049:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040018:0:112049:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040030:0:112049:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to regular blp list, 3 blwis in pool 10000000:01000000:0.0:1788572585.040034:0:112049:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:0.0:1788572585.040053:0:112018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1788572585.040059:0:112018:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040067:0:112018:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f5400 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:0.0:1788572585.040070:0:112018:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040094:0:112018:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572585.040102:0:112018:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572585.040108:0:112018:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572585.040110:0:112018:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:0.0:1788572585.040112:0:112018:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:2.0:1788572585.040295:0:112019:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788572585.040303:0:112019:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040310:0:112019:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367400 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572585.040313:0:112019:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040321:0:112019:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572585.040328:0:112019:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572585.040334:0:112019:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:2.0:1788572585.040336:0:112019:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:2.0:1788572585.040337:0:112019:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:2.0:1788572585.040369:0:112019:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040375:0:112019:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572585.040397:0:112019:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040404:0:112019:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc70367400/0x58a604dcef352c66 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931f0 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040422:0:112019:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572585.040425:0:112019:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040431:0:112019:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc702f4800 already unused, calling callback (ffffffffc1228c20) 10000000:00010000:2.0:1788572585.040434:0:112019:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040441:0:112019:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572585.040447:0:112019:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572585.040452:0:112019:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:2.0:1788572585.040454:0:112019:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:2.0:1788572585.040456:0:112019:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:2.0:1788572585.040467:0:112019:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040473:0:112019:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572585.040479:0:112019:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040484:0:112019:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f4800/0x58a604dcef352c6d lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931f7 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572585.040494:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572585.040496:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572585.040694:0:112019:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572585.040704:0:111977:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572585.040741:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:c95aecce-2623-4667-b83f-7aa0f7bacb74:2396:1875449863950592:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572585.040848:0:2396:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572585.040856:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:c95aecce-2623-4667-b83f-7aa0f7bacb74:2396:1875449863950720:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788572585.040933:0:112018:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040940:0:112018:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572585.040953:0:112018:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040959:0:112018:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f5400/0x58a604dcef352c35 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931e9 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.040972:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.040974:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.040976:0:112018:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572585.041525:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c95aecce-2623-4667-b83f-7aa0f7bacb74:2394:1875449863950848:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572585.044693:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:c95aecce-2623-4667-b83f-7aa0f7bacb74:2396:1875449863950720:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572585.044710:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:c95aecce-2623-4667-b83f-7aa0f7bacb74:2396:1875449863950592:192.168.202.132@tcp:103:ldlm_bl.0 00020000:01000000:0.0:1788572585.054994:0:112049:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff92fc67146800 active: 1 00020000:01000000:0.0:1788572585.055005:0:112049:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff92fc67142000 active: 1 00020000:01000000:0.0:1788572585.055009:0:112049:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:0.0:1788572585.055014:0:112049:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff92fc704fb000 00020000:01000000:0.0:1788572585.055017:0:112049:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc704fb000: disconnecting target lustre-OST0001-osc-ffff92fc704fb000 00000100:00080000:0.0:1788572585.055032:0:112049:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-OST0001_UUID 00000100:00080000:0.0:1788572585.055162:0:112049:0:(import.c:65:import_set_state_nolock()) ffff92fc4990b000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572585.055397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863950976:192.168.202.132@tcp:9:umount.0 00000100:00080000:1.0:1788572585.075687:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990b000 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572585.075712:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863950976:192.168.202.132@tcp:9:umount.0 00000100:00100000:1.0:1788572585.075792:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c95aecce-2623-4667-b83f-7aa0f7bacb74:2394:1875449863950848:192.168.202.132@tcp:103:ldlm_bl.0 00000100:00080000:0.0:1788572585.075931:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572585.075936:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc704fb000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572585.075941:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572585.076093:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c97 00000020:00000080:0.0:1788572585.076111:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc67142000/LOV_OSC_UUID 00000020:01000000:0.0:1788572585.076171:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff92fc704fb000 (flags='') 00000020:00000080:0.0:1788572585.076177:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572585.076185:0:112049:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff92fc704fb000: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572585.076190:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff92fc70033380) has no exports 00000020:00000080:3.0:1788572585.076677:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67142000/LOV_OSC_UUID for lustre-OST0001-osc-ffff92fc704fb000 00000008:01000000:0.0:1788572585.076902:0:112049:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff92fc704fb000: client import never connected 00000100:00080000:0.0:1788572585.076907:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788572585.076909:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc704fb000: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572585.076914:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572585.077467:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990b000 for lustre-OST0001-osc-ffff92fc704fb000 00000020:00000080:0.0:1788572585.077584:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572585.077595:0:112049:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:00000080:0.0:1788572585.077601:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc4990a000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:0.0:1788572585.077604:0:112049:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990a000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-OST0001-osc-ffff92fc704fb000 00000020:01000000:0.0:1788572585.077613:0:112049:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff92fc704fb000 (802259b2-6b09-4a33-8d4c-554d26b1ec81) 00020000:01000000:0.0:1788572585.077853:0:112049:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff92fc704fb000 00020000:01000000:0.0:1788572585.077856:0:112049:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff92fc704fb000: disconnecting target lustre-OST0000-osc-ffff92fc704fb000 00000100:00080000:0.0:1788572585.077867:0:112049:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-OST0000_UUID 00000100:00080000:0.0:1788572585.077978:0:112049:0:(import.c:65:import_set_state_nolock()) ffff92fc4990d800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572585.078052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863951104:192.168.202.132@tcp:9:umount.0 00000100:00080000:2.0:1788572585.083348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990d800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572585.083375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863951104:192.168.202.132@tcp:9:umount.0 00000100:00080000:0.0:1788572585.083441:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572585.083445:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc704fb000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572585.083450:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572585.083592:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c90 00000020:00000080:0.0:1788572585.083610:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc67146800/LOV_OSC_UUID 00000020:01000000:0.0:1788572585.083660:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff92fc704fb000 (flags='') 00000020:00000080:0.0:1788572585.083667:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788572585.083675:0:112049:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff92fc704fb000: forcing exports to disconnect: 0/3 00000020:00080000:0.0:1788572585.083679:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff92fc70030000) has no exports 00000020:00000080:3.0:1788572585.084115:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67146800/LOV_OSC_UUID for lustre-OST0000-osc-ffff92fc704fb000 00000008:01000000:0.0:1788572585.085386:0:112049:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff92fc704fb000: client import never connected 00000100:00080000:0.0:1788572585.085393:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:0.0:1788572585.085396:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc704fb000: committing for last_committed 0 gen 3 00000100:00100000:0.0:1788572585.085402:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572585.085615:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572585.085624:0:112049:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:00000080:0.0:1788572585.085629:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c89 00000020:00000080:0.0:1788572585.085637:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc67147000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000100:00080000:0.0:1788572585.085705:0:112049:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-MDT0000_UUID 00010000:00010000:0.0:1788572585.085798:0:112049:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572585.085846:0:112049:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 1/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572585.085858:0:112049:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572585.085874:0:112049:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc50cd0e40 going to sleep for 0 seconds 00000100:00100000:0.0:1788572585.085895:0:112049:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job umount:802259b2-6b09-4a33-8d4c-554d26b1ec81:112049:1875449863951232:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572585.085905:0:112049:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572585.085912:0:112049:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572585.085916:0:112049:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc704fb000 lock: ffff92fc70204a00/0x58a604dcef352ce4 lrc: 0/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217932c9 expref: -99 pid: 112041 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000020:00000080:3.0:1788572585.087193:0:40043:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990d800 for lustre-OST0000-osc-ffff92fc704fb000 00000020:00000080:3.0:1788572585.087206:0:40043:0:(genops.c:885:class_export_put()) final put ffff92fc4990f800/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:3.0:1788572585.087208:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990f800/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-OST0000-osc-ffff92fc704fb000 00000020:01000000:3.0:1788572585.087214:0:40043:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff92fc704fb000 (802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:00000080:3.0:1788572585.087466:0:40043:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67147000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-clilov-ffff92fc704fb000 00000100:00080000:0.0:1788572585.091202:0:112049:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc6800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788572585.091396:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863951360:192.168.202.132@tcp:39:umount.0 00000100:00080000:2.0:1788572585.094123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc6800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788572585.094158:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863951360:192.168.202.132@tcp:39:umount.0 00000100:00080000:0.0:1788572585.094267:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788572585.094271:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc704fb000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788572585.094280:0:112049:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67048380 x1875449863951360/t4294983784(4294983784) o39->lustre-MDT0000-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572595 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572585.094300:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572585.101431:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c7b 00000020:00000080:0.0:1788572585.101467:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc51c44000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000100:00080000:1.0:1788572585.101989:0:112049:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 802259b2-6b09-4a33-8d4c-554d26b1ec81->lustre-MDT0001_UUID 00000100:00080000:1.0:1788572585.103232:0:112049:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a800 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572585.103684:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863951488:192.168.202.132@tcp:39:umount.0 00000020:00000080:2.0:1788572585.107634:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51c44000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-MDT0000-mdc-ffff92fc704fb000 00000100:00080000:3.0:1788572585.113464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4990a800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788572585.113496:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:802259b2-6b09-4a33-8d4c-554d26b1ec81:2392:1875449863951488:192.168.202.132@tcp:39:umount.0 00000100:00080000:1.0:1788572585.114115:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572585.114120:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc704fb000: committing for last_committed 4294976775 gen 2 00000100:00100000:1.0:1788572585.114131:0:112049:0:(client.c:3164:ptlrpc_free_committed()) @@@ free request with old gen req@ffff92fc67288a80 x1875449863951488/t4294976777(4294976777) o39->lustre-MDT0001-mdc-ffff92fc704fb000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572595 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572585.114155:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572585.118638:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c82 00000020:00000080:1.0:1788572585.118654:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc51c40000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:1.0:1788572585.118770:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c74 00000020:00000080:1.0:1788572585.118775:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc51c43000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:2.0:1788572585.129036:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51c40000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-MDT0001-mdc-ffff92fc704fb000 00000020:01000000:1.0:1788572585.129466:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff92fc704fb000 (flags='') 00000020:00000080:1.0:1788572585.129477:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572585.129484:0:112049:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff92fc704fb000: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788572585.129489:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff92fc700719c0) has no exports 00000020:00000080:1.0:1788572585.129553:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572585.129560:0:112049:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:00000080:1.0:1788572585.129565:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc67147800/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:1.0:1788572585.129568:0:112049:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67147800/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-clilov-ffff92fc704fb000 00000020:01000000:1.0:1788572585.129573:0:112049:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff92fc704fb000 (802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:1.0:1788572585.129973:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff92fc704fb000 (flags='') 00000020:00000080:1.0:1788572585.129977:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572585.129981:0:112049:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff92fc704fb000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572585.129984:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff92fc70070000) has no exports 00000020:00000080:1.0:1788572585.130022:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572585.130026:0:112049:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:1.0:1788572585.130029:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff92fc704fb000 (flags='') 00000020:00000080:1.0:1788572585.130032:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572585.130035:0:112049:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff92fc704fb000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572585.130037:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff92fc70034d40) has no exports 00000020:00000080:2.0:1788572585.130568:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc51c43000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-clilmv-ffff92fc704fb000 00000020:00000080:2.0:1788572585.130575:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc67140000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:2.0:1788572585.130577:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc67140000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-clilmv-ffff92fc704fb000 00000020:01000000:2.0:1788572585.130603:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff92fc704fb000 (802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000008:01000000:1.0:1788572585.130852:0:112049:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff92fc704fb000: client import never connected 00000100:00080000:1.0:1788572585.130857:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788572585.130862:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc704fb000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788572585.130869:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788572585.137202:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572585.137216:0:112049:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000020:01000000:1.0:1788572585.137222:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff92fc704fb000 (flags='') 00000020:00000080:1.0:1788572585.137226:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572585.137230:0:112049:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff92fc704fb000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788572585.137234:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff92fc700319c0) has no exports 00000020:00000080:3.0:1788572585.137888:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc48dc6800 for lustre-MDT0000-mdc-ffff92fc704fb000 00000020:00000080:3.0:1788572585.137898:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc48dc4800/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:3.0:1788572585.137902:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc48dc4800/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-MDT0000-mdc-ffff92fc704fb000 00000020:01000000:3.0:1788572585.137913:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff92fc704fb000 (802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000008:01000000:1.0:1788572585.139363:0:112049:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff92fc704fb000: client import never connected 00000100:00080000:1.0:1788572585.139371:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788572585.139376:0:112049:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc704fb000: committing for last_committed 4294976775 gen 3 00000100:00100000:1.0:1788572585.139386:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788572585.147983:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:0.0:1788572585.147994:0:112049:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff92fc704fb000 (uuid 802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000080:02000400:0.0:1788572585.147999:0:112049:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:0.0:1788572585.148002:0:112049:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:0.0:1788572585.148044:0:112049:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff92fc704fb000 00000100:00080000:0.0:1788572585.148052:0:112049:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import c95aecce-2623-4667-b83f-7aa0f7bacb74->MGS 00010000:00010000:0.0:1788572585.148067:0:112049:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788572585.148693:0:112049:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788572585.148708:0:112049:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:0.0:1788572585.148711:0:112049:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:0.0:1788572585.148714:0:112049:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:0.0:1788572585.148784:0:112049:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.148791:0:112049:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:0.0:1788572585.148807:0:112049:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc50cd0e40 going to sleep for 0 seconds 00000100:00100000:0.0:1788572585.148818:0:112049:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job umount:c95aecce-2623-4667-b83f-7aa0f7bacb74:112049:1875449863951616:192.168.202.132@tcp:103:umount.0 00010000:00010000:0.0:1788572585.148827:0:112049:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:0.0:1788572585.148835:0:112049:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:0.0:1788572585.148839:0:112049:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.132@tcp lock: ffff92fc702f5c00/0x58a604dcef352c2e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0xc9a239a2217931e2 expref: -99 pid: 111965 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788572585.148868:0:112049:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0c000 MGS: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788572585.148956:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c95aecce-2623-4667-b83f-7aa0f7bacb74:2392:1875449863951744:192.168.202.132@tcp:251:umount.0 00000020:00000080:2.0:1788572585.149894:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc4990a800 for lustre-MDT0001-mdc-ffff92fc704fb000 00000020:00000080:2.0:1788572585.149911:0:31488:0:(genops.c:885:class_export_put()) final put ffff92fc4990c000/802259b2-6b09-4a33-8d4c-554d26b1ec81 00000020:00000080:2.0:1788572585.149916:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc4990c000/802259b2-6b09-4a33-8d4c-554d26b1ec81 for lustre-MDT0001-mdc-ffff92fc704fb000 00000020:01000000:2.0:1788572585.149924:0:31488:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff92fc704fb000 (802259b2-6b09-4a33-8d4c-554d26b1ec81) 00000100:00080000:1.0:1788572585.156061:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc42c0c000 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:1.0:1788572585.156073:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788572585.156474:0:2392:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788572585.156478:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572585.156499:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:c95aecce-2623-4667-b83f-7aa0f7bacb74:2392:1875449863951744:192.168.202.132@tcp:251:umount.0 00000100:00080000:2.0:1788572585.156559:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:2.0:1788572585.156563:0:112049:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:2.0:1788572585.156566:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572585.156568:0:112049:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:2.0:1788572585.156579:0:112049:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x58a604dcef352c27 00000020:00000080:2.0:1788572585.156587:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc42c0d800/c95aecce-2623-4667-b83f-7aa0f7bacb74 00000020:01000000:2.0:1788572585.156608:0:112049:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.132@tcp (flags='') 00000020:00000080:2.0:1788572585.156614:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572585.156629:0:112049:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.132@tcp: forcing exports to disconnect: 0/3 00000020:00080000:2.0:1788572585.156634:0:112049:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff92fc48cc99c0) has no exports 10000000:01000000:1.0:1788572585.156780:0:111980:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 00000020:00000080:1.0:1788572585.156919:0:31488:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc42c0d800/c95aecce-2623-4667-b83f-7aa0f7bacb74 for MGC192.168.202.132@tcp 10000000:01000000:1.0:1788572585.156975:0:112049:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.132@tcp: client import never connected 00000100:00080000:1.0:1788572585.156978:0:112049:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572585.156981:0:112049:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572585.156983:0:112049:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572585.156984:0:112049:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572585.157006:0:31488:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff92fc42c0c000 for MGC192.168.202.132@tcp 00000020:00000080:1.0:1788572585.157048:0:112049:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572585.157053:0:112049:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.132@tcp (uuid c95aecce-2623-4667-b83f-7aa0f7bacb74) 00000020:00000080:1.0:1788572585.157057:0:112049:0:(genops.c:885:class_export_put()) final put ffff92fc704fa000/c95aecce-2623-4667-b83f-7aa0f7bacb74 00000020:00000080:1.0:1788572585.157059:0:112049:0:(genops.c:844:class_export_destroy()) destroying export ffff92fc704fa000/c95aecce-2623-4667-b83f-7aa0f7bacb74 for MGC192.168.202.132@tcp 00000020:01000000:1.0:1788572585.157063:0:112049:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.132@tcp (c95aecce-2623-4667-b83f-7aa0f7bacb74) 00010000:00010000:2.0:1788572585.161764:0:111978:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:2.0:1788572585.163450:0:112019:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:1.0:1788572585.164653:0:111977:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00010000:00010000:1.0:1788572585.166958:0:112018:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-4 blwis in pool) 00000100:00100000:1.0:1788572585.167337:0:111967:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff92fc53d1a600:111967] 0 exiting: rc = 1 00000100:00100000:1.0:1788572585.167406:0:111968:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff92fc53d1a800:111968] 1 exiting: rc = 0 00000100:00100000:2.0:1788572585.167657:0:111969:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff92fc58ca2e00:111969] 0 exiting: rc = 1 00000100:00100000:2.0:1788572585.170807:0:111970:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff92fc491eca00:111970] 1 exiting: rc = 0 00000100:00100000:2.0:1788572585.171001:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.171013:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572585.171464:0:111972:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff92fc491ed200:111972] 1 exiting: rc = 0 00000100:00100000:0.0:1788572585.172275:0:111971:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff92fc491ede00:111971] 0 exiting: rc = 1 00000100:00100000:0.0:1788572585.172342:0:111973:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff92fc412e7e00:111973] 2 exiting: rc = 0 00000100:00100000:2.0:1788572585.173475:0:111974:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff92fc4328ac00:111974] 0 exiting: rc = 1 00000100:00100000:2.0:1788572585.174281:0:111976:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff92fc613aae00:111976] 2 exiting: rc = 0 00000100:00100000:2.0:1788572585.174353:0:111975:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff92fc470f3600:111975] 1 exiting: rc = 0 00000100:00100000:2.0:1788572585.174514:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174520:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174525:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174531:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174536:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174541:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174546:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174551:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174558:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174563:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174568:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174575:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174580:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174585:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174589:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174594:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174599:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174605:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174610:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174616:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174750:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174756:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174763:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174768:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174774:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174779:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174783:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174788:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174793:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174799:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174805:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174811:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174817:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174822:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174827:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174832:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174836:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174842:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174846:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174851:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174857:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174863:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174869:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174875:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174891:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174896:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174901:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174906:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174911:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174915:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174920:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174925:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174931:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174937:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174942:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174948:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174953:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174958:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174962:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174967:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174971:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174977:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174982:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174988:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.174995:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175002:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175007:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175012:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175017:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175022:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175029:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175034:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175040:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175045:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175050:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175056:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175061:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175066:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175071:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175077:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175083:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175089:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175094:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175099:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175105:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175110:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175116:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175121:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175440:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175451:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175458:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175464:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175471:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175477:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175483:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175489:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175497:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175503:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175509:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175514:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175520:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175524:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175529:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175534:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175538:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175544:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175549:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175555:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175561:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175566:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175571:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175578:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175583:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175588:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175593:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175599:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175604:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175609:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.175614:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176718:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176745:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176751:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176757:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176762:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176768:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176774:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176780:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:2.0:1788572585.176785:0:112049:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:2.0:1788572585.191031:0:112049:0:(obd_mount.c:710:lustre_put_lsi()) put ffff92fc704fb000 1 00000020:01000004:2.0:1788572585.191039:0:112049:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff92fc704f9800 00000020:01000004:2.0:1788572585.902143:0:112066:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:2.0:1788572585.902149:0:112066:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:2.0:1788572585.902150:0:112066:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.132@tcp:/lustre/d60.sanity-sec 00000020:01000004:2.0:1788572585.902152:0:112066:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock,test_dummy_encryption 00000080:01200004:2.0:1788572585.902259:0:112066:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff92fc67229000 00000080:01000004:2.0:1788572585.902269:0:112066:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:2.0:1788572585.902285:0:112066:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.132@tcp' 00000020:01000004:2.0:1788572585.902314:0:112066:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.132@tcp (typ=mgc) 00000020:00000080:2.0:1788572585.902325:0:112066:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.132@tcp (ffff92fc48cc99c0) 00000020:00000080:2.0:1788572585.902337:0:112066:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:2.0:1788572585.904772:0:112066:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788572585.904799:0:112066:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:2.0:1788572585.904991:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:2.0:1788572585.904999:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:2.0:1788572585.906595:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 1 min 2 max 34 00000100:00100000:2.0:1788572585.906605:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_001' 00000100:00100000:1.0:1788572585.909508:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:1.0:1788572585.909573:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:1.0:1788572585.926603:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:0.0:1788572585.926747:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:1.0:1788572585.928184:0:112066:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572585.929004:0:112066:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_canceld: allocate 64 new 8192-byte reqbufs (0/64 left), rc = 0 00000100:00100000:1.0:1788572585.929201:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 0 min 3 max 34 00000100:00100000:1.0:1788572585.929208:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_000' 00000100:00100000:0.0:1788572585.931111:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 1 min 3 max 34 00000100:00100000:0.0:1788572585.931121:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_001' 00000100:00100000:1.0:1788572585.931558:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:1.0:1788572585.931565:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:0.0:1788572585.932009:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:0.0:1788572585.932016:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:1.0:1788572585.932490:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:1.0:1788572585.932496:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00000100:00100000:0.0:1788572585.934818:0:112066:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 2 min 3 max 34 00000100:00100000:0.0:1788572585.934832:0:112066:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_002' 00010000:00010000:1.0:1788572585.937216:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572585.937222:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572585.937224:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.938127:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.938131:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572585.938133:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:0.0:1788572585.941303:0:112066:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc4536c800@MGC192.168.202.132@tcp: add connection 192.168.202.132@tcp at head 00010000:00010000:0.0:1788572585.941476:0:112066:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.132@tcp-0 is initialized 00000040:01000000:0.0:1788572585.941483:0:112066:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.132@tcp ctxt 1 is initialized 10000000:01000000:0.0:1788572585.942883:0:112080:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:1.0:1788572585.944119:0:112066:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.132@tcp (uuid b62659d6-7f40-4480-a04f-ba5899daf967) 00000020:00000080:1.0:1788572585.944155:0:112066:0:(genops.c:1385:class_connect()) connect: client b62659d6-7f40-4480-a04f-ba5899daf967, cookie 0x58a604dcef352cf2 00000100:00080000:1.0:1788572585.944171:0:112066:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:1.0:1788572585.944183:0:112066:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572585.944223:0:112066:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449863951872/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572585.944255:0:112066:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import b62659d6-7f40-4480-a04f-ba5899daf967->MGS 00000080:00200000:1.0:1788572585.944280:0:112066:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff92fc67229000 (sb ffff92fc67229000) 00000080:01000000:1.0:1788572585.944393:0:112066:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock,test_dummy_encryption 00000080:00000004:1.0:1788572585.944395:0:112066:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:1.0:1788572585.944400:0:112066:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:00000004:1.0:1788572585.944401:0:112066:0:(llite_lib.c:1242:ll_options()) next opt=test_dummy_encryption 00000080:01000000:1.0:1788572585.944409:0:112066:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: a74e4397-7fb3-41e0-aa97-867f5c09bb42 10000000:01000000:1.0:1788572585.944894:0:112066:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:1.0:1788572585.944898:0:112066:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff92fc67229000 10000000:01000000:1.0:1788572585.944902:0:112066:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff92fc48cc99c0 10000000:01000000:1.0:1788572585.944906:0:112066:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 00000100:00100000:0.0:1788572585.945467:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863951872:192.168.202.132@tcp:250:mount.lustre.0 00010000:00010000:0.0:1788572585.945740:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788572585.946200:0:112066:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff92fc48cc99c0 from 1 10000000:01000000:1.0:1788572585.946205:0:112066:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788572585.946244:0:112066:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572585.946255:0:112066:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572585.946266:0:112066:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788572585.946286:0:112066:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67288700 x1875449863952000/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572585.946295:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc608486c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572585.951119:0:2392:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.132@tcp: committing for last_committed 0 gen 1 00000100:00080000:0.0:1788572585.951136:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.132@tcp: connect to target with instance 0 00000100:00080000:0.0:1788572585.951146:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.132@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:0.0:1788572585.951152:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:0.0:1788572585.951157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:0.0:1788572585.951161:0:2392:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788572585.951173:0:2392:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:0.0:1788572585.951188:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000d60ba37e): req@ffff92fc67288700 x1875449863952000/t0(0) o101->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572585.951212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863951872:192.168.202.132@tcp:250:mount.lustre.0 00000100:00100000:0.0:1788572585.951418:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc608486c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572585.962211:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952000:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788572585.962243:0:112066:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932d7 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572585.962257:0:112066:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217932d7 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572585.962339:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952128:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788572585.962428:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.970059:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952128:192.168.202.132@tcp:501:mount.lustre.0 10000000:01000000:3.0:1788572585.970106:0:112066:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:3.0:1788572585.970131:0:112066:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932d7 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.970143:0:112066:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc702f5a00/0x58a604dcef352cf9 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932d7 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572585.970154:0:112066:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff92fc67229000 10000000:01000000:3.0:1788572585.970160:0:112066:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:3.0:1788572585.970164:0:112066:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff92fc67229000 10000000:01000000:3.0:1788572585.970166:0:112066:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:3.0:1788572585.970170:0:112066:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff92fc67229000 10000000:01000000:3.0:1788572585.970173:0:112066:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:3.0:1788572585.970176:0:112066:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff92fc67229000 from 1 10000000:01000000:3.0:1788572585.970179:0:112066:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:3.0:1788572585.970218:0:112066:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.970227:0:112066:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.970240:0:112066:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.970261:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952256:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572585.970332:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45182480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572585.987919:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952256:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572585.988022:0:112066:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932de expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572585.988048:0:112066:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217932de expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572585.988150:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952384:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572585.988224:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45182480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572585.998268:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952384:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572585.998350:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952512:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572585.998441:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc430ae780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.004376:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863952512:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572586.004959:0:112084:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112084:1875449863952640:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572586.005054:0:112084:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45182480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.012007:0:112084:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112084:1875449863952640:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572586.012058:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572586.012070:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.012072:0:112084:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:3.0:1788572586.012080:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff92fc67229000 00000020:00000080:3.0:1788572586.012083:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572586.012100:0:112084:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff92fc67229000 (ffff92fc70070000) 00000020:00000080:3.0:1788572586.012118:0:112084:0:(obd_config.c:720:class_attach_name()) OBD: dev 1 attached type lov with refcount 2 00000020:01000000:3.0:1788572586.012124:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff92fc67229000 00000020:00000080:3.0:1788572586.012127:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788572586.012384:0:112084:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff92fc67229000 (uuid a74e4397-7fb3-41e0-aa97-867f5c09bb42) 00000020:01000000:3.0:1788572586.012389:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572586.012392:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.012394:0:112084:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:3.0:1788572586.012397:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572586.012400:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.012402:0:112084:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:3.0:1788572586.012406:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff92fc67229000 00000020:00000080:3.0:1788572586.012408:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572586.012414:0:112084:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff92fc67229000 (ffff92fc700719c0) 00000020:00000080:3.0:1788572586.012423:0:112084:0:(obd_config.c:720:class_attach_name()) OBD: dev 2 attached type lmv with refcount 2 00000020:01000000:3.0:1788572586.012428:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff92fc67229000 00000020:00000080:3.0:1788572586.012430:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788572586.012579:0:112084:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff92fc67229000 (uuid a74e4397-7fb3-41e0-aa97-867f5c09bb42) 00000020:01000000:3.0:1788572586.012583:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572586.013109:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.013113:0:112084:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:3.0:1788572586.013118:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572586.013122:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.013124:0:112084:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:3.0:1788572586.013127:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:3.0:1788572586.013129:0:112084:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:3.0:1788572586.013132:0:112084:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:3.0:1788572586.013137:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572586.013139:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.013140:0:112084:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:3.0:1788572586.013142:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572586.013146:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.013148:0:112084:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:3.0:1788572586.013151:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788572586.013155:0:112084:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:3.0:1788572586.013179:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff92fc67229000 00000020:00000080:3.0:1788572586.013182:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788572586.013191:0:112084:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff92fc67229000 (ffff92fc70073380) 00000020:00000080:3.0:1788572586.013201:0:112084:0:(obd_config.c:720:class_attach_name()) OBD: dev 3 attached type mdc with refcount 2 00000020:01000000:3.0:1788572586.013206:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff92fc67229000 00000020:00000080:3.0:1788572586.013209:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788572586.013273:0:112084:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc48dc5800@lustre-MDT0000-mdc-ffff92fc67229000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572586.023621:0:112084:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff92fc67229000-1 is initialized 00000040:01000000:2.0:1788572586.023895:0:112084:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff92fc67229000 ctxt 13 is initialized 00000020:00000080:2.0:1788572586.028246:0:112084:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff92fc67229000 (uuid a74e4397-7fb3-41e0-aa97-867f5c09bb42) 00000020:01000000:2.0:1788572586.028264:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc67229000 00000020:00000080:2.0:1788572586.028269:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572586.028300:0:112084:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:2.0:1788572586.029680:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.029694:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.029696:0:112084:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:2.0:1788572586.029703:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572586.029706:0:112084:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:2.0:1788572586.029708:0:112084:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572586.029709:0:112084:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572586.029711:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572586.029713:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.029714:0:112084:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572586.029717:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572586.029740:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.029742:0:112084:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:2.0:1788572586.029747:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572586.029753:0:112084:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572586.029778:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff92fc67229000 00000020:00000080:2.0:1788572586.029781:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572586.029795:0:112084:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff92fc67229000 (ffff92fc48cc8000) 00000020:00000080:2.0:1788572586.029815:0:112084:0:(obd_config.c:720:class_attach_name()) OBD: dev 4 attached type mdc with refcount 2 00000020:01000000:2.0:1788572586.029821:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff92fc67229000 00000020:00000080:2.0:1788572586.029824:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572586.031176:0:112084:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5824c800@lustre-MDT0001-mdc-ffff92fc67229000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572586.041080:0:112084:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff92fc67229000-2 is initialized 00000040:01000000:2.0:1788572586.042145:0:112084:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff92fc67229000 ctxt 13 is initialized 00000020:00000080:2.0:1788572586.049504:0:112084:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff92fc67229000 (uuid a74e4397-7fb3-41e0-aa97-867f5c09bb42) 00000020:01000000:2.0:1788572586.049522:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff92fc67229000 00000020:00000080:2.0:1788572586.049527:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:2.0:1788572586.049554:0:112084:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:2.0:1788572586.049598:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.049602:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.049604:0:112084:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:2.0:1788572586.049607:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572586.049610:0:112084:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:2.0:1788572586.049612:0:112084:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572586.049614:0:112084:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572586.049616:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572586.049618:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.049619:0:112084:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572586.049621:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572586.049626:0:112084:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.202.132@tcp:/lustre/d60.sanity-sec 00000020:00000080:2.0:1788572586.049630:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.049631:0:112084:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:2.0:1788572586.049634:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572586.049640:0:112084:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572586.049666:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff92fc67229000 00000020:00000080:2.0:1788572586.049669:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572586.049680:0:112084:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff92fc67229000 (ffff92fc48cccd40) 00000020:00000080:2.0:1788572586.049697:0:112084:0:(obd_config.c:720:class_attach_name()) OBD: dev 5 attached type osc with refcount 2 00000020:01000000:2.0:1788572586.049704:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff92fc67229000 00000020:00000080:2.0:1788572586.049706:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572586.049797:0:112084:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc5824d000@lustre-OST0000-osc-ffff92fc67229000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572586.050147:0:112084:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff92fc67229000-3 is initialized 00000020:00000080:2.0:1788572586.051760:0:112084:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff92fc67229000 (uuid a74e4397-7fb3-41e0-aa97-867f5c09bb42) 00000020:01000000:2.0:1788572586.051779:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc67229000 00000020:00000080:2.0:1788572586.051783:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572586.051809:0:112084:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:2.0:1788572586.051818:0:112084:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc67229000: Adding target lustre-OST0000_UUID 00020000:01000000:2.0:1788572586.051824:0:112084:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:2.0:1788572586.051838:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.051841:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.051854:0:112084:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:2.0:1788572586.051859:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:2.0:1788572586.051862:0:112084:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:2.0:1788572586.051864:0:112084:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572586.051866:0:112084:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:2.0:1788572586.051867:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:2.0:1788572586.051870:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.051872:0:112084:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:2.0:1788572586.051874:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:2.0:1788572586.051879:0:112084:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.202.132@tcp:/lustre/d60.sanity-sec 00000020:00000080:2.0:1788572586.051882:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.051883:0:112084:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:2.0:1788572586.051886:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:2.0:1788572586.051890:0:112084:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.202.132@tcp to nid 0x20000c0a8ca84 (192.168.202.132@tcp) 00000020:01000000:2.0:1788572586.053673:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff92fc67229000 00000020:00000080:2.0:1788572586.053678:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:2.0:1788572586.053691:0:112084:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff92fc67229000 (ffff92fc48ccb380) 00000020:00000080:2.0:1788572586.053711:0:112084:0:(obd_config.c:720:class_attach_name()) OBD: dev 6 attached type osc with refcount 2 00000020:01000000:2.0:1788572586.053717:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff92fc67229000 00000020:00000080:2.0:1788572586.053743:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:2.0:1788572586.053838:0:112084:0:(ldlm_lib.c:102:import_set_conn()) imp ffff92fc58248000@lustre-OST0001-osc-ffff92fc67229000: add connection 192.168.202.132@tcp at head 00010000:00010000:2.0:1788572586.055225:0:112084:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff92fc67229000-4 is initialized 00000020:00000080:2.0:1788572586.055568:0:112084:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff92fc67229000 (uuid a74e4397-7fb3-41e0-aa97-867f5c09bb42) 00000020:01000000:2.0:1788572586.055985:0:112084:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff92fc67229000 00000020:00000080:2.0:1788572586.055992:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:2.0:1788572586.056014:0:112084:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:2.0:1788572586.056022:0:112084:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff92fc67229000: Adding target lustre-OST0001_UUID 00020000:01000000:2.0:1788572586.056027:0:112084:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:2.0:1788572586.056038:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.056041:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.056044:0:112084:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:2.0:1788572586.056047:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572586.056050:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.056052:0:112084:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:2.0:1788572586.056055:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:2.0:1788572586.056057:0:112084:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:2.0:1788572586.056059:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.056061:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.056063:0:112084:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:2.0:1788572586.056112:0:112084:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112084:1875449863952768:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572586.056267:0:112084:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f3e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.064952:0:112084:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112084:1875449863952768:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572586.065002:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572586.065012:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.065015:0:112084:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:2.0:1788572586.065021:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:2.0:1788572586.065063:0:112084:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112084:1875449863952896:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572586.065159:0:112084:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f3e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.072641:0:112084:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112084:1875449863952896:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572586.072690:0:112084:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572586.072698:0:112084:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572586.072701:0:112084:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:3.0:1788572586.072705:0:112084:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:3.0:1788572586.075643:0:112066:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:3.0:1788572586.075675:0:112066:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572586.075702:0:112066:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932de expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.075715:0:112066:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70205c00/0x58a604dcef352d00 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932de expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572586.075739:0:112066:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff92fc67229000 from 1 10000000:01000000:3.0:1788572586.075742:0:112066:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788572586.075788:0:112066:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.075800:0:112066:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.075810:0:112066:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572586.075840:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953024:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572586.076072:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.079180:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953024:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572586.079208:0:112066:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932e5 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.079223:0:112066:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217932e5 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572586.079970:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953152:192.168.202.132@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788572586.080055:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.084156:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953152:192.168.202.132@tcp:256:mount.lustre.0 10000000:01000000:3.0:1788572586.085076:0:112066:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:3.0:1788572586.085140:0:112066:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932e5 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.085153:0:112066:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70204e00/0x58a604dcef352d31 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932e5 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572586.085229:0:112066:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff92fc67229000 from 1 10000000:01000000:2.0:1788572586.085233:0:112066:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572586.085281:0:112066:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.085290:0:112066:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.085299:0:112066:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.085326:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953280:192.168.202.132@tcp:101:mount.lustre.0 00000100:00100000:2.0:1788572586.085410:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.088237:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953280:192.168.202.132@tcp:101:mount.lustre.0 00010000:00010000:2.0:1788572586.088268:0:112066:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932ec expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.088281:0:112066:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xc9a239a2217932ec expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.088377:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953408:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572586.088437:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.094293:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953408:192.168.202.132@tcp:501:mount.lustre.0 00000100:00100000:2.0:1788572586.094388:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953536:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572586.094496:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.096498:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:b62659d6-7f40-4480-a04f-ba5899daf967:112066:1875449863953536:192.168.202.132@tcp:503:mount.lustre.0 00000100:00100000:2.0:1788572586.097478:0:112087:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112087:1875449863953664:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572586.097667:0:112087:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f36c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.099686:0:112087:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112087:1875449863953664:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572586.099817:0:112087:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572586.099828:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.099830:0:112087:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:2.0:1788572586.099838:0:112087:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc67229000 00000020:00000080:2.0:1788572586.099840:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572586.099859:0:112087:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.099862:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.099865:0:112087:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:2.0:1788572586.099868:0:112087:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572586.099870:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.099872:0:112087:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:2.0:1788572586.099875:0:112087:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc67229000 00000020:00000080:2.0:1788572586.099877:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:2.0:1788572586.100038:0:112087:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.100043:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.100044:0:112087:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000020:01000000:2.0:1788572586.100047:0:112087:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:2.0:1788572586.100050:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.100051:0:112087:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) general nodemap..fileset 00000020:01000000:2.0:1788572586.100054:0:112087:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff92fc67229000 00000020:00000080:2.0:1788572586.100056:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:00020000:2.0:1788572586.101211:0:112087:0:(obd_config.c:1432:process_param2_config()) lctl: error invoking upcall /usr/sbin/lctl set_param nodemap..fileset=/c0dir: rc = -2; time 1148us 00000100:00100000:2.0:1788572586.101263:0:112087:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112087:1875449863953792:192.168.202.132@tcp:502:llog_process_th.0 00000100:00100000:2.0:1788572586.101362:0:112087:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f36c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.109887:0:112087:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:b62659d6-7f40-4480-a04f-ba5899daf967:112087:1875449863953792:192.168.202.132@tcp:502:llog_process_th.0 00000020:01000000:2.0:1788572586.110686:0:112087:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:2.0:1788572586.110698:0:112087:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:2.0:1788572586.110701:0:112087:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) general nodemap..fileset 00000040:00080000:2.0:1788572586.110707:0:112087:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 10 count 10 00000020:01000000:3.0:1788572586.113612:0:112066:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-9 (rc=0) 10000000:01000000:3.0:1788572586.113645:0:112066:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.132@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788572586.113675:0:112066:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932ec expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.113689:0:112066:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.132@tcp lock: ffff92fc70366a00/0x58a604dcef352d38 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0xc9a239a2217932ec expref: -99 pid: 112066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:3.0:1788572586.113710:0:112066:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:3.0:1788572586.113755:0:112066:0:(genops.c:1385:class_connect()) connect: client a74e4397-7fb3-41e0-aa97-867f5c09bb42, cookie 0x58a604dcef352d3f 00800000:01000000:3.0:1788572586.113777:0:112066:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect a74e4397-7fb3-41e0-aa97-867f5c09bb42 to lustre-clilmv-ffff92fc67229000 00800000:01000000:3.0:1788572586.113783:0:112066:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff92fc67229000(a74e4397-7fb3-41e0-aa97-867f5c09bb42) - lustre-MDT0000_UUID, a74e4397-7fb3-41e0-aa97-867f5c09bb42 00000020:00000080:3.0:1788572586.113790:0:112066:0:(genops.c:1385:class_connect()) connect: client a74e4397-7fb3-41e0-aa97-867f5c09bb42, cookie 0x58a604dcef352d46 00000100:00080000:3.0:1788572586.113801:0:112066:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572586.113813:0:112066:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572586.113827:0:112066:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572586.113866:0:112066:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863953920/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572586.113893:0:112066:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID 00000100:00100000:2.0:1788572586.114095:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863953920:192.168.202.132@tcp:38:mount.lustre.0 00800000:01000000:3.0:1788572586.114317:0:112066:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff92fc67229000(a74e4397-7fb3-41e0-aa97-867f5c09bb42) successfully (5) 00800000:00000004:3.0:1788572586.114321:0:112066:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff92fc67229000: age 3341, max_age 0 00800000:01000000:3.0:1788572586.114365:0:112066:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff92fc67229000(a74e4397-7fb3-41e0-aa97-867f5c09bb42) - lustre-MDT0001_UUID, a74e4397-7fb3-41e0-aa97-867f5c09bb42 00000020:00000080:3.0:1788572586.114372:0:112066:0:(genops.c:1385:class_connect()) connect: client a74e4397-7fb3-41e0-aa97-867f5c09bb42, cookie 0x58a604dcef352d4d 00000100:00080000:3.0:1788572586.114378:0:112066:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572586.114383:0:112066:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572586.114389:0:112066:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572586.114402:0:112066:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863954176/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572586.114415:0:112066:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID 00800000:01000000:3.0:1788572586.114459:0:112066:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff92fc67229000(a74e4397-7fb3-41e0-aa97-867f5c09bb42) successfully (5) 00800000:00000004:3.0:1788572586.114462:0:112066:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff92fc67229000: age 3341, max_age 0 00000080:00000004:3.0:1788572586.114478:0:112066:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff92fc67229000: age 3341, max_age 4340 00800000:00000004:3.0:1788572586.114483:0:112066:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff92fc67229000: age 3341, max_age 4340 00000100:00080000:3.0:1788572586.114500:0:112066:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67339f80 x1875449863954432/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572586.114509:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5c00 going to sleep for 0 seconds 00000100:00080000:3.0:1788572586.114537:0:2395:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6733b100 x1875449863954304/t0(0) o41->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572586.114549:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1788572586.114553:0:2395:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc67338380 x1875449863954048/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572586.122146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863954176:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572586.124767:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc67229000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572586.124791:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff92fc67229000: connect to target with instance 1 00000100:00080000:3.0:1788572586.124803:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff92fc67229000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572586.124819:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:3.0:1788572586.124824:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:1.0:1788572586.126927:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000086b30de6): req@ffff92fc67339f80 x1875449863954432/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572586.126948:0:2392:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a8793aa6): req@ffff92fc67338380 x1875449863954048/t0(0) o41->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572586.126961:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863953920:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:1.0:1788572586.127016:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff92fc67229000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788572586.127026:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff92fc67229000: connect to target with instance 1 00000100:00080000:1.0:1788572586.127037:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff92fc67229000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:1.0:1788572586.127051:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:1.0:1788572586.127055:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:3.0:1788572586.127276:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.129069:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863954176:192.168.202.132@tcp:38:mount.lustre.0 00000100:00100000:2.0:1788572586.146604:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:0:1875449863954048:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:2.0:1788572586.146632:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:0:1875449863954304:192.168.202.132@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572586.146836:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863954432:192.168.202.132@tcp:41:mount.lustre.0 00800000:00000004:3.0:1788572586.146867:0:112066:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff92fc67229000: update ffff92fc700737d8 cache blocks 317009/353529 objects 1023700/1024000 00000080:00000004:3.0:1788572586.146878:0:112066:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff92fc67229000: update ffff92fc70071e18 cache blocks 317009/353529 objects 1023700/1024000 00000100:00100000:3.0:1788572586.146938:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863954560:192.168.202.132@tcp:40:mount.lustre.0 00000100:00100000:3.0:1788572586.147016:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d50c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.155519:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863954560:192.168.202.132@tcp:40:mount.lustre.0 00000080:00000004:3.0:1788572586.155547:0:112066:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000422:0x1:0x0] 00000080:00100000:3.0:1788572586.155562:0:112066:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 2146304 00020000:01000000:3.0:1788572586.155567:0:112066:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:3.0:1788572586.155592:0:112066:0:(genops.c:1385:class_connect()) connect: client a74e4397-7fb3-41e0-aa97-867f5c09bb42, cookie 0x58a604dcef352d54 00000020:00000080:3.0:1788572586.155610:0:112066:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352d5b 00000100:00080000:3.0:1788572586.155617:0:112066:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572586.155625:0:112066:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572586.155637:0:112066:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572586.155663:0:112066:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449863954688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572586.155756:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863954688:192.168.202.132@tcp:8:mount.lustre.0 00000100:00080000:3.0:1788572586.156023:0:112066:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID 00020000:01000000:3.0:1788572586.156043:0:112066:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff92fc67229000) active 00000020:00000080:3.0:1788572586.156088:0:112066:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x58a604dcef352d62 00000100:00080000:3.0:1788572586.156094:0:112066:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 0 00000100:00080000:3.0:1788572586.156101:0:112066:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572586.156107:0:112066:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572586.156120:0:112066:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863954816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572586.165118:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863954816:192.168.202.132@tcp:8:mount.lustre.0 00000100:00080000:3.0:1788572586.166789:0:112066:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID 00020000:01000000:3.0:1788572586.166810:0:112066:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff92fc67229000) active 00000080:02000400:3.0:1788572586.166837:0:112066:0:(llite_lib.c:801:client_common_fill_super()) Test dummy encryption mode enabled 00000100:00100000:3.0:1788572586.166919:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863954944:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788572586.166992:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d50c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.172746:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff92fc67229000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572586.172765:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff92fc67229000: connect to target with instance 1 00000100:00080000:3.0:1788572586.172777:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff92fc67229000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572586.172791:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:3.0:1788572586.172795:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572586.172802:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:3.0:1788572586.172806:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0467af2e3640038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572586.172810:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572586.172825:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863954688:192.168.202.132@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572586.174485:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863954944:192.168.202.132@tcp:900:mount.lustre.0 00000100:00100000:3.0:1788572586.174560:0:112066:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863955072:192.168.202.132@tcp:33:mount.lustre.0 00000100:00100000:3.0:1788572586.174688:0:112066:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d50c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.177415:0:112066:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112066:1875449863955072:192.168.202.132@tcp:33:mount.lustre.0 00000080:00200000:3.0:1788572586.177621:0:112066:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00200000:3.0:1788572586.177632:0:112066:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000100:00100000:3.0:1788572586.185798:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff92fc67229000: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572586.185816:0:2392:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff92fc67229000: connect to target with instance 1 00000100:00080000:3.0:1788572586.185826:0:2392:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff92fc67229000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572586.185850:0:2392:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:3.0:1788572586.185854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00020000:01000000:3.0:1788572586.185863:0:2392:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0001_UUID active 00000080:00000004:3.0:1788572586.185867:0:2392:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572586.185873:0:2392:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:3.0:1788572586.185886:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863954816:192.168.202.132@tcp:8:mount.lustre.0 00000080:02000400:2.0:1788572586.365353:0:112066:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.58_2_gfa0a981 00000080:00000004:2.0:1788572586.365360:0:112066:0:(super25.c:182:lustre_fill_super()) 192.168.202.132@tcp:/lustre/d60.sanity-sec: Mount complete 00000080:00200008:2.0:1788572586.617513:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dd348), request_mask 606, flags 0, foreign 0 00000080:00200000:2.0:1788572586.617527:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:2.0:1788572586.617554:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.617617:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.617627:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.617638:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.617665:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955200:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572586.617789:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.625490:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955200:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572586.625517:0:112108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625529:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625541:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625554:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625567:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625580:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625588:0:112108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572586.625599:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617ce00 x1875449863955200/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572586.625612:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.625625:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02000 type 13 mode 4 res 8589935650/1 (0 0) 00000080:00200000:2.0:1788572586.625631:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:2.0:1788572586.625691:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955328:192.168.202.132@tcp:900:ls.0 00000100:00100000:2.0:1788572586.625787:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.627896:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955328:192.168.202.132@tcp:900:ls.0 00000100:00100000:2.0:1788572586.627959:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955456:192.168.202.132@tcp:33:ls.0 00000100:00100000:2.0:1788572586.628030:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.637795:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955456:192.168.202.132@tcp:33:ls.0 00000080:00200000:3.0:1788572586.637932:0:112108:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000422:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1788572586.637945:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:3.0:1788572586.637953:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572586.637957:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bcb8 00010000:00010000:3.0:1788572586.637962:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.637976:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00010000:3.0:1788572586.638139:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getxattr flags 0 00010000:00010000:3.0:1788572586.638197:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.638206:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.638216:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 3/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572586.638243:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955584:192.168.202.132@tcp:101:ls.0 00000100:00100000:3.0:1788572586.638326:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d59c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.641048:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955584:192.168.202.132@tcp:101:ls.0 00010000:00010000:3.0:1788572586.641078:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.641094:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.641105:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.641123:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc67339c00 x1875449863955584/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 568/1040 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572586.641134:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.641147:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02000 type 13 mode 4 res 8589935650/1 (0 0) 00000080:00200000:3.0:1788572586.641154:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00010000:3.0:1788572586.641177:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d70 00000080:00010000:3.0:1788572586.641186:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d70 from it ffff9e038236bb80 00010000:00010000:3.0:1788572586.641190:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.641202:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572586.646331:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:3.0:1788572586.646361:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.646378:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.646394:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572586.646401:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.646407:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572586.646411:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:3.0:1788572586.646420:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bbb0 00010000:00010000:3.0:1788572586.646423:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.646434:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:3.0:1788572586.646451:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 44 00000080:00200008:3.0:1788572586.646460:0:112108:0:(file.c:1090:ll_file_open()) START file /:[0x200000422:0x1:0x0](ffff92fc665dd348), flags 304000 00000080:00200008:3.0:1788572586.646471:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dd348), request_mask 2047, flags 0, foreign 0 00000002:00010000:3.0:1788572586.646487:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: readdir flags 0 00010000:00010000:3.0:1788572586.646494:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572586.646503:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000100:00100000:3.0:1788572586.647555:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955712:192.168.202.132@tcp:37:ls.0 00000100:00100000:3.0:1788572586.647717:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d59c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.658512:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955712:192.168.202.132@tcp:37:ls.0 00010000:00010000:3.0:1788572586.659251:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:3.0:1788572586.659272:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572586.659332:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:3.0:1788572586.659352:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.659366:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.659379:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572586.659388:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.659394:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572586.659398:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:3.0:1788572586.659403:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bcd8 00010000:00010000:3.0:1788572586.659406:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.659413:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200008:3.0:1788572586.659505:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000422:0x1:0x0](ffff92fc665dd348), request_mask 256, flags 0, foreign 0 00000080:00200000:3.0:1788572586.659511:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:3.0:1788572586.659520:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.659528:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572586.659537:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572586.659544:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.659548:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572586.659551:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:3.0:1788572586.659555:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bcb8 00010000:00010000:3.0:1788572586.659558:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572586.659565:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572586.659618:0:112108:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348) pos/size0/4096 32bit_api 0 00000080:00200000:3.0:1788572586.659681:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:3.0:1788572586.659689:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.659696:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572586.659705:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572586.659712:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.659716:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572586.659718:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:3.0:1788572586.659755:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bb58 00010000:00010000:3.0:1788572586.659759:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572586.659767:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572586.659777:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:3.0:1788572586.659791:0:112108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file1, dir=[0x200000422:0x1:0x0](ffff92fc665dd348), flags=8 00000080:00200000:3.0:1788572586.659795:0:112108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file1, dir=[0x200000422:0x1:0x0](ffff92fc665dd348), intent=getattr 00000002:00010000:3.0:1788572586.659815:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 2l”vä¡6Èpë¥Òr¸,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572586.659876:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.659885:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.659894:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572586.659918:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955840:192.168.202.132@tcp:101:ls.0 00000100:00100000:3.0:1788572586.659992:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d59c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.663778:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863955840:192.168.202.132@tcp:101:ls.0 00010000:00010000:3.0:1788572586.663812:0:112108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663846:0:112108:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663853:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663864:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663874:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663892:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663901:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663908:0:112108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.663919:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67339c00 x1875449863955840/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/672 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572586.663930:0:112108:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572586.663942:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.663954:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02000 type 13 mode 4 res 8589935650/2 (0 0) 00000080:00200000:3.0:1788572586.663959:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572586.663994:0:112108:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dba88) 00000080:00200000:3.0:1788572586.664090:0:112108:0:(file.c:6843:ll_layout_conf()) [0x200000422:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788572586.664105:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) for lock 0x58a604dcef352d77 00000080:00010000:3.0:1788572586.664118:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572586.664122:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d77 from it ffff9e038236bb20 00010000:00010000:3.0:1788572586.664125:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.664140:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572586.664181:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file file1:[0x200000422:0x2:0x0](ffff92fc665dba88), request_mask 606, flags 0, foreign 0 00000080:00200000:3.0:1788572586.664189:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dba88),name=file1 00000002:00010000:3.0:1788572586.664223:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x2:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.664232:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.664243:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572586.664250:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.664255:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) 00000080:00010000:3.0:1788572586.664258:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) for lock 0x58a604dcef352d77 00000080:00010000:3.0:1788572586.664262:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d77 from it ffff9e038236bcb8 00010000:00010000:3.0:1788572586.664265:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572586.664272:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572586.664292:0:112108:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788572586.664301:0:112108:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000422:0x2:0x0] 00000008:00010000:3.0:1788572586.664352:0:112108:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572586.664356:0:112108:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc7049d5b0, osc lock ffff92fc4645cd10, flags 0x400001002 00010000:00010000:3.0:1788572586.664405:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 3/1,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.664416:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 3/1,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788572586.664449:0:112108:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788572586.664476:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863955968:192.168.202.132@tcp:101:ls.0 00010000:00010000:3.0:1788572586.667117:0:2396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.667144:0:2396:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: --/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.667159:0:2396:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.667174:0:2396:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788572586.667189:0:2396:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2, setting rss=2; kms=2, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788572586.667201:0:2396:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788572586.667220:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.667234:0:2396:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.667241:0:2396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.667250:0:2396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788572586.667269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863955968:192.168.202.132@tcp:101:ls.0 00000080:00200000:0.0:1788572586.667352:0:112108:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000422:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:0.0:1788572586.667358:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc67174250 (ffff92fc6713b128/0) at cl_lock_release():245 00000020:00010000:0.0:1788572586.667361:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67174250 (ffff92fc6713b128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.667365:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049d5b0 (ffff92fc6713b3f8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.667369:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67174250 (ffff92fc6713b128/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572586.667372:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049d5b0 (ffff92fc6713b128/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572586.667387:0:112108:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572586.667434:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:0.0:1788572586.667457:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572586.667475:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572586.667488:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788572586.667495:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572586.667501:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:0.0:1788572586.667504:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:0.0:1788572586.667509:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bbf8 00010000:00010000:0.0:1788572586.667512:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788572586.667519:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788572586.667531:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572586.667538:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file1, flags=9 00000080:00200000:0.0:1788572586.667583:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:0.0:1788572586.667592:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572586.667599:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572586.667608:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788572586.667613:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572586.667617:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:0.0:1788572586.667620:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:0.0:1788572586.667624:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bb58 00010000:00010000:0.0:1788572586.667627:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:0.0:1788572586.667634:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788572586.667643:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572586.667656:0:112108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subdir, dir=[0x200000422:0x1:0x0](ffff92fc665dd348), flags=8 00000080:00200000:0.0:1788572586.667660:0:112108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subdir, dir=[0x200000422:0x1:0x0](ffff92fc665dd348), intent=getattr 00000002:00010000:0.0:1788572586.667685:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: Ì™C¸ æºî×Wè9f,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572586.668001:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.668010:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.668025:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572586.668046:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956096:192.168.202.132@tcp:101:ls.0 00000100:00100000:0.0:1788572586.668120:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.672160:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956096:192.168.202.132@tcp:101:ls.0 00010000:00010000:3.0:1788572586.672193:0:112108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672217:0:112108:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672226:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672238:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672250:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672267:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672276:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672284:0:112108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.672296:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc67048700 x1875449863956096/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572586.672309:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.672320:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc47f02000 type 13 mode 4 res 9663677451/1 (0 0) 00000100:00100000:3.0:1788572586.672378:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956224:192.168.202.132@tcp:900:ls.0 00000100:00100000:3.0:1788572586.672454:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.678472:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956224:192.168.202.132@tcp:900:ls.0 00000002:00010000:2.0:1788572586.680677:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572586.680774:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.680785:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.680795:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.680831:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956352:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572586.681001:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.690503:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956352:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572586.690533:0:112108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 4/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690546:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690568:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690584:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690602:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690610:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690617:0:112108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572586.690636:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc4617d500 x1875449863956352/t0(0) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/616 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572586.690654:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690668:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245800 type 13 mode 4 res 9663677451/1 (0 0) 00000080:00200000:2.0:1788572586.690687:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572586.690759:0:112108:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88) 00000080:00200000:2.0:1788572586.690766:0:112108:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040b:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1788572586.690781:0:112108:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) for remote lock 0x58a604dcef352d8c 00000080:00010000:2.0:1788572586.690785:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352d85 00000080:00010000:2.0:1788572586.690794:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572586.690798:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d85 from it ffff9e038236bb20 00010000:00010000:2.0:1788572586.690801:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690811:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788572586.690834:0:112108:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x58a604dcef352d8c from it ffff9e038236bb20 00010000:00010000:2.0:1788572586.690837:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.690845:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572586.690878:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file subdir:[0x24000040b:0x1:0x0](ffff92fc665e3a88), request_mask 606, flags 0, foreign 0 00000080:00200000:2.0:1788572586.690884:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88),name=subdir 00000002:00010000:2.0:1788572586.690922:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.690932:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572586.690944:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572586.690951:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.690956:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572586.690958:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352d8c 00000080:00010000:2.0:1788572586.690962:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d8c from it ffff9e038236bcb8 00010000:00010000:2.0:1788572586.690964:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572586.690971:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572586.691330:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:2.0:1788572586.691342:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.691354:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572586.691363:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572586.691376:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.691380:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:2.0:1788572586.691383:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:2.0:1788572586.691387:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bbf8 00010000:00010000:2.0:1788572586.691390:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572586.691398:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572586.691407:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:2.0:1788572586.691413:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=9 00000080:00200000:2.0:1788572586.691433:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:2.0:1788572586.691439:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.691445:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572586.691453:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572586.691458:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.691461:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:2.0:1788572586.691464:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:2.0:1788572586.691466:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bbf8 00010000:00010000:2.0:1788572586.691469:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572586.691480:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572586.691487:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:2.0:1788572586.691490:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=9 00000002:00010000:2.0:1788572586.691504:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1788572586.691541:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.691548:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.691566:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 3/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.691588:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956480:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572586.691673:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.695193:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956480:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572586.695222:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.695239:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.695251:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572586.695301:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff92fc4617d500 x1875449863956480/t0(0) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 568/1048 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572586.695313:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.695338:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245800 type 13 mode 4 res 9663677451/1 (0 0) 00000080:00200000:2.0:1788572586.695344:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 128 on i1 [0x24000040b:0x1:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1788572586.695370:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352d93 00000080:00010000:2.0:1788572586.695379:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d93 from it ffff9e038236bb80 00010000:00010000:2.0:1788572586.695384:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.695394:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572586.695460:0:112108:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:2.0:1788572586.695537:0:112108:0:(file.c:479:ll_file_release()) START file /:[0x200000422:0x1:0x0](ffff92fc665dd348), flags 304000 00000080:00200000:2.0:1788572586.699810:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:2.0:1788572586.699857:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.699879:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572586.699896:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572586.699905:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.699913:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:2.0:1788572586.699917:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:2.0:1788572586.699925:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bbf8 00010000:00010000:2.0:1788572586.699929:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572586.699938:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788572586.699955:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:2.0:1788572586.699963:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=267 00000080:00200000:2.0:1788572586.699970:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 44 00000080:00200008:2.0:1788572586.699978:0:112108:0:(file.c:1090:ll_file_open()) START file subdir:[0x24000040b:0x1:0x0](ffff92fc665e3a88), flags 304000 00000002:00010000:2.0:1788572586.699994:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:2.0:1788572586.700258:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.700267:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572586.700280:0:112108:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc67229000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572586.700293:0:112108:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617d500 x1875449863956736/t0(0) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/131800 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572586.700303:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.700339:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956736:192.168.202.132@tcp:101:ls.0 00000100:00100000:2.0:1788572586.700540:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572586.704793:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956736:192.168.202.132@tcp:101:ls.0 00010000:00010000:2.0:1788572586.704831:0:112108:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704844:0:112108:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704852:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704859:0:112108:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 3/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704867:0:112108:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 4/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704873:0:112108:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:2.0:1788572586.704878:0:112108:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 4/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704884:0:112108:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 4/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704891:0:112108:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 5/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704897:0:112108:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572586.704904:0:112108:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 3/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.704910:0:112108:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352d9a lrc: 0/0,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572586.704940:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617d500 x1875449863956736/t4294976778(4294976778) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'ls.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572586.704953:0:112108:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d500 x1875449863956736/t4294976778(4294976778) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'ls.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572586.704966:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572586.704986:0:112108:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x24000040b:0x1:0x0], updating i_size 4096 00000002:00100000:2.0:1788572586.705017:0:112108:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617d500 x1875449863956736/t4294976778(4294976778) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'ls.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572586.705084:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file subdir:[0x24000040b:0x1:0x0](ffff92fc665e3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572586.705091:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88),name=subdir 00000002:00010000:2.0:1788572586.705131:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.705145:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572586.705157:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:2.0:1788572586.705165:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.705170:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572586.705173:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352d8c 00000080:00010000:2.0:1788572586.705179:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d8c from it ffff9e038236bcd8 00010000:00010000:2.0:1788572586.705182:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:2.0:1788572586.705190:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200008:2.0:1788572586.705286:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file subdir:[0x24000040b:0x1:0x0](ffff92fc665e3a88), request_mask 256, flags 0, foreign 0 00000080:00200000:2.0:1788572586.705291:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88),name=subdir 00000002:00010000:2.0:1788572586.705306:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.705327:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572586.705337:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572586.705343:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.705348:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) 00000080:00010000:2.0:1788572586.705350:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x1:0x0](ffff92fc665e3a88) for lock 0x58a604dcef352d8c 00000080:00010000:2.0:1788572586.705354:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d8c from it ffff9e038236bcb8 00010000:00010000:2.0:1788572586.705357:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572586.705365:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572586.705397:0:112108:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88) pos/size0/4096 32bit_api 0 00000002:00010000:0.0:1788572586.734635:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x1:0x0]) in obj [0x24000040b:0x1:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788572586.734663:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572586.734682:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000100:00100000:0.0:1788572586.735466:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956864:192.168.202.132@tcp:37:ls.0 00000100:00100000:0.0:1788572586.735665:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572586.749192:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956864:192.168.202.132@tcp:37:ls.0 00010000:00010000:1.0:1788572586.749959:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572586.749976:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367000/0x58a604dcef352d8c lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793371 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572586.750068:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:1.0:1788572586.750094:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572586.750112:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572586.750126:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572586.750134:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572586.750140:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:1.0:1788572586.750144:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:1.0:1788572586.750150:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bb58 00010000:00010000:1.0:1788572586.750154:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572586.750161:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572586.750172:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:1.0:1788572586.750179:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=24 00000080:00200000:1.0:1788572586.750185:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572586.750197:0:112108:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=subfile1, dir=[0x24000040b:0x1:0x0](ffff92fc665e3a88), flags=8 00000080:00200000:1.0:1788572586.750201:0:112108:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=subfile1, dir=[0x24000040b:0x1:0x0](ffff92fc665e3a88), intent=getattr 00000002:00010000:1.0:1788572586.750229:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: D¨æ5*XdQ>\K7„=¿,[0x0:0x0:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572586.750278:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.750287:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.750297:0:112108:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112108 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572586.750323:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956992:192.168.202.132@tcp:101:ls.0 00000100:00100000:1.0:1788572586.750391:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc53faee40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572586.762131:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863956992:192.168.202.132@tcp:101:ls.0 00010000:00010000:1.0:1788572586.762159:0:112108:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 4/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762178:0:112108:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762187:0:112108:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762198:0:112108:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762214:0:112108:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762226:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762235:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762247:0:112108:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572586.762260:0:112108:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff92fc6728b100 x1875449863956992/t0(0) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/672 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'ls.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572586.762272:0:112108:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572586.762283:0:112108:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762294:0:112108:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff92fc43245800 type 13 mode 4 res 9663677451/2 (0 0) 00000080:00200000:1.0:1788572586.762300:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x24000040b:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572586.762344:0:112108:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d0908) 00000080:00200000:1.0:1788572586.762446:0:112108:0:(file.c:6843:ll_layout_conf()) [0x24000040b:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572586.762461:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) for lock 0x58a604dcef352da1 00000080:00010000:1.0:1788572586.762473:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) 00000080:00010000:1.0:1788572586.762477:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352da1 from it ffff9e038236bb20 00010000:00010000:1.0:1788572586.762480:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.762490:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572586.762527:0:112108:0:(file.c:6296:ll_getattr_dentry()) START file subfile1:[0x24000040b:0x2:0x0](ffff92fc665d0908), request_mask 606, flags 0, foreign 0 00000080:00200000:1.0:1788572586.763115:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d0908),name=subfile1 00000002:00010000:1.0:1788572586.763146:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x2:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572586.763159:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572586.763171:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572586.763177:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x24000040b:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572586.763183:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) 00000080:00010000:1.0:1788572586.763186:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) for lock 0x58a604dcef352da1 00000080:00010000:1.0:1788572586.763191:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352da1 from it ffff9e038236bcb8 00010000:00010000:1.0:1788572586.763194:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572586.763202:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572586.763221:0:112108:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788572586.763230:0:112108:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040b:0x2:0x0] 00000008:00010000:1.0:1788572586.763262:0:112108:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572586.763266:0:112108:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4630, osc lock ffff92fc47ff5430, flags 0x400001002 00010000:00010000:1.0:1788572586.763328:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.763338:0:112108:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572586.763382:0:112108:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788572586.763404:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572586.763712:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863957120:192.168.202.132@tcp:101:ls.0 00010000:00010000:0.0:1788572586.766624:0:2394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.766652:0:2394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.766668:0:2394:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.766683:0:2394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788572586.766699:0:2394:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2, setting rss=2; kms=2, end=18446744073709551615 ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788572586.766717:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788572586.766755:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.766765:0:2394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.766774:0:2394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.766783:0:2394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788572586.766801:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863957120:192.168.202.132@tcp:101:ls.0 00000080:00200000:3.0:1788572586.767022:0:112108:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040b:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:3.0:1788572586.767029:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49debc20 (ffff92fc49fcee18/0) at cl_lock_release():245 00000020:00010000:3.0:1788572586.767033:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49debc20 (ffff92fc49fcee18/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572586.767039:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4630 (ffff92fc49fce2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572586.767043:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49debc20 (ffff92fc49fcee18/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572586.767047:0:112108:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4630 (ffff92fc49fcee18/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572586.767060:0:112108:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572586.767110:0:112108:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:3.0:1788572586.767131:0:112108:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.767149:0:112108:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788572586.767162:0:112108:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1788572586.767169:0:112108:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.767177:0:112108:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:3.0:1788572586.767180:0:112108:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:3.0:1788572586.767185:0:112108:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e038236bbf8 00010000:00010000:3.0:1788572586.767190:0:112108:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1788572586.767198:0:112108:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788572586.767209:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:3.0:1788572586.767216:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=25 00000080:00200000:3.0:1788572586.767222:0:112108:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 1 00000080:00200000:3.0:1788572586.767225:0:112108:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subfile1, flags=9 00000080:00200000:3.0:1788572586.767278:0:112108:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200008:3.0:1788572586.767299:0:112108:0:(file.c:479:ll_file_release()) START file subdir:[0x24000040b:0x1:0x0](ffff92fc665e3a88), flags 304000 00000002:00100000:3.0:1788572586.767368:0:112108:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617d500 x1875449863956736/t4294976778(4294976778) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572602 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'ls.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572586.767380:0:112108:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc67229000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572586.767385:0:112108:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67339c00 x1875449863957376/t0(0) o35->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572586.767414:0:112108:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863957376:192.168.202.132@tcp:35:ls.0 00000100:00100000:3.0:1788572586.767487:0:112108:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.770496:0:112108:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ls:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112108:1875449863957376:192.168.202.132@tcp:35:ls.0 00000002:00100000:3.0:1788572586.770532:0:112108:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617d500 x1875449863956736/t4294976778(4294976778) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'ls.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572586.780990:0:112114:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:0.0:1788572586.781021:0:112114:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572586.781042:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788572586.781057:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788572586.781066:0:112114:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572586.781073:0:112114:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:0.0:1788572586.781078:0:112114:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:0.0:1788572586.781086:0:112114:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e03832bfbf8 00010000:00010000:0.0:1788572586.781090:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788572586.781099:0:112114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788572586.781272:0:112114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:0.0:1788572586.781281:0:112114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file1, flags=265 00000080:00200000:0.0:1788572586.781288:0:112114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dba88), inode mode 81a4 mask 44 00000080:00200008:0.0:1788572586.781295:0:112114:0:(file.c:1090:ll_file_open()) START file file1:[0x200000422:0x2:0x0](ffff92fc665dba88), flags 100000 00000002:00010000:0.0:1788572586.781460:0:112114:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x2:0x0]) in obj [0x200000422:0x1:0x0], intent: open flags 0240100001 00010000:00010000:0.0:1788572586.781514:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112114 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.781522:0:112114:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112114 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572586.781533:0:112114:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc67229000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572586.781547:0:112114:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67048000 x1875449863957632/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 760/131968 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572586.781624:0:112114:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112114 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572586.781649:0:112114:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112114:1875449863957632:192.168.202.132@tcp:101:cat.0 00000100:00100000:0.0:1788572586.782121:0:112114:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572586.789571:0:112114:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff92fc67229000: committing for last_committed 4294983785 gen 1 00000100:00100000:1.0:1788572586.789591:0:112114:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff92fc67048000 x1875449863957632/t4294983785(4294983785) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572602 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572586.789605:0:112114:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112114:1875449863957632:192.168.202.132@tcp:101:cat.0 00010000:00010000:1.0:1788572586.789632:0:112114:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 4/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789649:0:112114:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789657:0:112114:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 4/1,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789671:0:112114:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789684:0:112114:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789697:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789706:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 5/2,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789713:0:112114:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 4/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572586.789898:0:112114:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc67048000 x1875449863957632/t4294983785(4294983785) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572586.789911:0:112114:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048000 x1875449863957632/t4294983785(4294983785) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 760/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572586.789928:0:112114:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572586.789941:0:112114:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789955:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572586.789967:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572586.789975:0:112114:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789983:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.789991:0:112114:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572586.790018:0:112114:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1788572586.790026:0:112114:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572586.790112:0:112114:0:(file.c:6843:ll_layout_conf()) [0x200000422:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788572586.790118:0:112114:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) for lock 0x58a604dcef352d77 00000080:00010000:1.0:1788572586.790125:0:112114:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d77 from it ffff9e03832bfbf0 00010000:00010000:1.0:1788572586.790129:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572586.790137:0:112114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000002:00100000:1.0:1788572586.790164:0:112114:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc67048000 x1875449863957632/t4294983785(4294983785) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572586.790245:0:112114:0:(file.c:6296:ll_getattr_dentry()) START file file1:[0x200000422:0x2:0x0](ffff92fc665dba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572586.790254:0:112114:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x2:0x0](ffff92fc665dba88),name=file1 00010000:00010000:0.0:1788572586.790565:0:112078:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572586.790574:0:112078:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790583:0:112078:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc51bc3600 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572586.790589:0:112078:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790598:0:112078:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790606:0:112078:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 3/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790625:0:112078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788572586.790660:0:112078:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 2/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790669:0:112078:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572586.790693:0:112078:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 1/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790701:0:112078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc3600/0x58a604dcef352daf lrc: 0/0,0 mode: --/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217933be expref: -99 pid: 112114 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.790717:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572586.790864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572586.790867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572586.790883:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863957760:192.168.202.132@tcp:103:ldlm_bl.0 00000002:00010000:1.0:1788572586.791435:0:112114:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x2:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572586.791449:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572586.791459:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788572586.791467:0:112114:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572586.791475:0:112114:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) 00000080:00010000:1.0:1788572586.791479:0:112114:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x2:0x0](ffff92fc665dba88) for lock 0x58a604dcef352d77 00000080:00010000:1.0:1788572586.791484:0:112114:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d77 from it ffff9e03832bfcd8 00010000:00010000:1.0:1788572586.791487:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572586.791495:0:112114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70205800/0x58a604dcef352d77 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a221793340 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572586.791517:0:112114:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788572586.791528:0:112114:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000422:0x2:0x0] 00000008:00010000:1.0:1788572586.791639:0:112114:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572586.791644:0:112114:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4130, osc lock ffff92fc47ff51d0, flags 0x400001002 00010000:00010000:1.0:1788572586.791664:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572586.791676:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788572586.791688:0:112114:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572586.791693:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572586.791701:0:112114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572586.791746:0:112114:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572586.791756:0:112114:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000422:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:1.0:1788572586.791761:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:1.0:1788572586.791764:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572586.791767:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4130 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572586.791771:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572586.791773:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4130 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572586.791784:0:112114:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788572586.791893:0:112114:0:(file.c:2483:do_file_read_iter()) START file file1:[0x200000422:0x2:0x0], ppos: 0, count: 4194304 00000080:00200000:1.0:1788572586.791915:0:112114:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:1.0:1788572586.791934:0:112114:0:(file.c:2090:ll_file_io_generic()) file1: read ppos: 0, bytes: 4194304 00000080:00200000:1.0:1788572586.791939:0:112114:0:(file.c:1848:ll_io_set_mirror()) file1: desiginated mirror: 0 00000020:00200000:1.0:1788572586.791945:0:112114:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b87d4759[0x110, 1, [0x200000422:0x2:0x0] hash] 00000020:00200000:1.0:1788572586.791947:0:112114:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:1.0:1788572586.791950:0:112114:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572586.791961:0:112114:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:1.0:1788572586.791966:0:112114:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572586.791978:0:112114:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:1.0:1788572586.791987:0:112114:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:1.0:1788572586.791990:0:112114:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4130, osc lock ffff92fc47ff51d0, flags 0x0 00010000:00010000:1.0:1788572586.791999:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572586.793015:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572586.793029:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572586.793037:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572586.793045:0:112114:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572586.793056:0:112114:0:(vvp_io.c:835:vvp_io_read_start()) file1: read [0, 4194304) 00000001:00010000:1.0:1788572586.793072:0:112114:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000422:0x2:0x0] 00000008:00010000:1.0:1788572586.793083:0:112114:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572586.793085:0:112114:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc464d4730, osc lock ffff92fc47ff4850, flags 0x400001002 00010000:00010000:1.0:1788572586.793096:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572586.793106:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572586.793114:0:112114:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788572586.793117:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1788572586.793125:0:112114:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000020:00200000:1.0:1788572586.793131:0:112114:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788572586.793137:0:112114:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000422:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:1.0:1788572586.793142:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:1.0:1788572586.793145:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788572586.793147:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4730 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788572586.793150:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:1.0:1788572586.793152:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4730 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:1.0:1788572586.793162:0:112114:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:1.0:1788572586.793207:0:112114:0:(osc_cache.c:3059:osc_queue_sync_pages()) ### no dlm lock: extent: ffff92fc42c26480 00000008:00010000:1.0:1788572586.793258:0:110544:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc42c26480 00010000:00010000:1.0:1788572586.793281:0:110544:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00000008:00100000:1.0:1788572586.795190:0:110544:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc67288000 - object 0x280000401:553 offset 0<>4096 00000100:00100000:1.0:1788572586.795269:0:2394:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572586.795291:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863957888:192.168.202.132@tcp:3:cat.0 00000008:00010000:1.0:1788572586.807447:0:2394:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc42c26480 00000020:00200000:1.0:1788572586.807464:0:2394:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788572586.807482:0:2394:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc42c26480 00000100:00100000:1.0:1788572586.807507:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863957888:192.168.202.132@tcp:3:cat.0 00000100:00100000:1.0:1788572586.807532:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863957760:192.168.202.132@tcp:103:ldlm_bl.0 00000020:00010000:0.0:1788572586.807885:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:0.0:1788572586.807891:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.807895:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc464d4130 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1788572586.807906:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 1 00010000:00010000:0.0:1788572586.807922:0:112114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:0.0:1788572586.807939:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572586.807942:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc464d4130 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572586.807950:0:112114:0:(vvp_io.c:533:vvp_io_advance()) advancing 2 bytes 00000080:00200000:0.0:1788572586.807965:0:112114:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572586.807970:0:112114:0:(file.c:2238:ll_file_io_generic()) file1: 1 io complete with rc: 0, result: 2, restart: 0 00000080:00200000:0.0:1788572586.807974:0:112114:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 2 00000080:00200008:0.0:1788572586.808013:0:112114:0:(file.c:2483:do_file_read_iter()) START file file1:[0x200000422:0x2:0x0], ppos: 2, count: 4194304 00000080:00200000:0.0:1788572586.808024:0:112114:0:(file.c:2090:ll_file_io_generic()) file1: read ppos: 2, bytes: 4194304 00000080:00200000:0.0:1788572586.808028:0:112114:0:(file.c:1848:ll_io_set_mirror()) file1: desiginated mirror: 0 00000020:00200000:0.0:1788572586.808034:0:112114:0:(cl_io.c:214:cl_io_rw_init()) header@00000000b87d4759[0x110, 2, [0x200000422:0x2:0x0] hash] 00000020:00200000:0.0:1788572586.808036:0:112114:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2, 4194306) 0 0 00000080:00200000:0.0:1788572586.808041:0:112114:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572586.808056:0:112114:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 2 chunk: [2, 4194306) 4194306, 4194304 00020000:00200000:0.0:1788572586.808060:0:112114:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572586.808075:0:112114:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:0.0:1788572586.808087:0:112114:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788572586.808091:0:112114:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc67240030, osc lock ffff92fc459337c0, flags 0x0 00010000:00010000:0.0:1788572586.808112:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808132:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808145:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808154:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808162:0:112114:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788572586.808170:0:112114:0:(vvp_io.c:835:vvp_io_read_start()) file1: read [2, 4194306) 00000001:00010000:0.0:1788572586.808184:0:112114:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000422:0x2:0x0] 00000008:00010000:0.0:1788572586.808190:0:112114:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572586.808192:0:112114:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc67240530, osc lock ffff92fc45933db0, flags 0x400001002 00010000:00010000:0.0:1788572586.808200:0:112114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808209:0:112114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:0.0:1788572586.808218:0:112114:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572586.808221:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 7/2,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808233:0:112114:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 6/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:0.0:1788572586.808239:0:112114:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572586.808244:0:112114:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000422:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:0.0:1788572586.808247:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:0.0:1788572586.808250:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.808252:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67240530 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.808255:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572586.808258:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67240530 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572586.808264:0:112114:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:2 00000020:00010000:0.0:1788572586.808269:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:0.0:1788572586.808271:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.808273:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67240030 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1788572586.808277:0:112114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 4/1,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.808284:0:112114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 000000007726d508/0x58a604dcef352d7e lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a221793347 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00010000:0.0:1788572586.808294:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572586.808296:0:112114:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67240030 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572586.808301:0:112114:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:0.0:1788572586.808306:0:112114:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572586.808369:0:112114:0:(file.c:2238:ll_file_io_generic()) file1: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:0.0:1788572586.808371:0:112114:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:0.0:1788572586.808451:0:112114:0:(file.c:479:ll_file_release()) START file file1:[0x200000422:0x2:0x0](ffff92fc665dba88), flags 100000 00000002:00100000:0.0:1788572586.808509:0:112114:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc67048000 x1875449863957632/t4294983785(4294983785) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572586.808520:0:112114:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff92fc67229000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572586.808525:0:112114:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc6704a680 x1875449863958144/t0(0) o35->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:23/10 lens 392/736 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572586.808543:0:112114:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112114:1875449863958144:192.168.202.132@tcp:35:cat.0 00000100:00100000:0.0:1788572586.808689:0:112114:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc45809300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572586.811520:0:112114:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112114:1875449863958144:192.168.202.132@tcp:35:cat.0 00000002:00100000:0.0:1788572586.811553:0:112114:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc67048000 x1875449863957632/t4294983785(4294983785) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572586.840095:0:112115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:2.0:1788572586.840126:0:112115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572586.840146:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572586.840162:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572586.840171:0:112115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572586.840179:0:112115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:2.0:1788572586.840183:0:112115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:2.0:1788572586.840191:0:112115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e0382ff3bf8 00010000:00010000:2.0:1788572586.840195:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572586.840204:0:112115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572586.840220:0:112115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:2.0:1788572586.840228:0:112115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subdir, flags=25 00000080:00200000:2.0:1788572586.840234:0:112115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x1:0x0](ffff92fc665e3a88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572586.840238:0:112115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=subfile1, flags=265 00000080:00200000:2.0:1788572586.840241:0:112115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d0908), inode mode 81a4 mask 44 00000080:00200008:2.0:1788572586.840247:0:112115:0:(file.c:1090:ll_file_open()) START file subfile1:[0x24000040b:0x2:0x0](ffff92fc665d0908), flags 100000 00000002:00010000:2.0:1788572586.840784:0:112115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x2:0x0]) in obj [0x24000040b:0x1:0x0], intent: open flags 0240100001 00010000:00010000:2.0:1788572586.840853:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572586.840862:0:112115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112115 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572586.840875:0:112115:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc67229000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572586.840888:0:112115:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc4617fb80 x1875449863958400/t0(0) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 656/131864 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572586.840898:0:112115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572586.840919:0:112115:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112115:1875449863958400:192.168.202.132@tcp:101:cat.0 00000100:00100000:2.0:1788572586.841097:0:112115:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc484f30c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.844778:0:112115:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112115:1875449863958400:192.168.202.132@tcp:101:cat.0 00010000:00010000:3.0:1788572586.844807:0:112115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 4/1,0 mode: --/CR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844838:0:112115:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844847:0:112115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 4/1,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844859:0:112115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844877:0:112115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844891:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844899:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 5/2,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844907:0:112115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 4/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572586.844919:0:112115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff92fc4617fb80 x1875449863958400/t4294976780(4294976780) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572586.844933:0:112115:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617fb80 x1875449863958400/t4294976780(4294976780) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 656/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572586.844946:0:112115:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572586.844960:0:112115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.844973:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572586.844987:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572586.844995:0:112115:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.845002:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.845020:0:112115:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572586.845038:0:112115:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1788572586.845045:0:112115:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x24000040b:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572586.845097:0:112115:0:(file.c:6843:ll_layout_conf()) [0x24000040b:0x2:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788572586.845103:0:112115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) for lock 0x58a604dcef352da1 00000080:00010000:3.0:1788572586.845110:0:112115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352da1 from it ffff9e0382ff3bf0 00010000:00010000:3.0:1788572586.845113:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572586.845122:0:112115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000002:00100000:3.0:1788572586.845151:0:112115:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff92fc4617fb80 x1875449863958400/t4294976780(4294976780) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788572586.845223:0:112115:0:(file.c:6296:ll_getattr_dentry()) START file subfile1:[0x24000040b:0x2:0x0](ffff92fc665d0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572586.845231:0:112115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x24000040b:0x2:0x0](ffff92fc665d0908),name=subfile1 00000002:00010000:3.0:1788572586.845265:0:112115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x24000040b:0x2:0x0]) in obj [0x24000040b:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572586.845275:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.845285:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572586.845292:0:112115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x24000040b:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572586.845297:0:112115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) 00000080:00010000:3.0:1788572586.845301:0:112115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x24000040b:0x2:0x0](ffff92fc665d0908) for lock 0x58a604dcef352da1 00000080:00010000:3.0:1788572586.845306:0:112115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352da1 from it ffff9e0382ff3cd8 00010000:00010000:0.0:1788572586.846242:0:112077:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572586.846253:0:112077:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846262:0:112077:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff92fc70367200 already unused, calling callback (ffffffffc19d9630) 00010000:00010000:0.0:1788572586.846267:0:112077:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846275:0:112077:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846285:0:112077:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 3/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846304:0:112077:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788572586.846368:0:112077:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 2/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846376:0:112077:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572586.846403:0:112077:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846411:0:112077:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367200/0x58a604dcef352db6 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a2217933d3 expref: -99 pid: 112115 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572586.846435:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572586.846437:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572586.846439:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572586.846448:0:2393:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788572586.846469:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863958528:192.168.202.132@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572586.861516:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 3/1,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.861532:0:112115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc702f4400/0x58a604dcef352da1 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xc9a239a2217933a9 expref: -99 pid: 112108 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788572586.861832:0:112115:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788572586.861844:0:112115:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040b:0x2:0x0] 00000008:00010000:0.0:1788572586.861866:0:112115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572586.861870:0:112115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc67240030, osc lock ffff92fc459337c0, flags 0x400001002 00010000:00010000:0.0:1788572586.861892:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572586.861909:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788572586.861922:0:112115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572586.861926:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572586.861935:0:112115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788572586.861946:0:112115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572586.861955:0:112115:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040b:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:0.0:1788572586.861959:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:0.0:1788572586.861962:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.861965:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67240030 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.861968:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572586.861971:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67240030 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572586.861981:0:112115:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788572586.862075:0:112115:0:(file.c:2483:do_file_read_iter()) START file subfile1:[0x24000040b:0x2:0x0], ppos: 0, count: 4194304 00000080:00200000:0.0:1788572586.862099:0:112115:0:(rw.c:1925:ll_readpage()) fast read pgno: 0 00000080:00200000:0.0:1788572586.862120:0:112115:0:(file.c:2090:ll_file_io_generic()) subfile1: read ppos: 0, bytes: 4194304 00000080:00200000:0.0:1788572586.862125:0:112115:0:(file.c:1848:ll_io_set_mirror()) subfile1: desiginated mirror: 0 00000020:00200000:0.0:1788572586.862131:0:112115:0:(cl_io.c:214:cl_io_rw_init()) header@0000000060999004[0x110, 1, [0x24000040b:0x2:0x0] hash] 00000020:00200000:0.0:1788572586.862134:0:112115:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [0, 4194304) 0 0 00000080:00200000:0.0:1788572586.862136:0:112115:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572586.862148:0:112115:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4194304) 4194304, 4194304 00020000:00200000:0.0:1788572586.862153:0:112115:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572586.862166:0:112115:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1023] 00000008:00010000:0.0:1788572586.862174:0:112115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1788572586.862177:0:112115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc67240030, osc lock ffff92fc459337c0, flags 0x0 00010000:00010000:0.0:1788572586.862186:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572586.862197:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4194303) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572586.862207:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572586.862215:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572586.862222:0:112115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572586.862228:0:112115:0:(vvp_io.c:835:vvp_io_read_start()) subfile1: read [0, 4194304) 00000001:00010000:0.0:1788572586.862238:0:112115:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040b:0x2:0x0] 00000008:00010000:0.0:1788572586.862243:0:112115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788572586.862244:0:112115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc67240530, osc lock ffff92fc45933db0, flags 0x400001002 00010000:00010000:0.0:1788572586.862250:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572586.862257:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788572586.862263:0:112115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788572586.862266:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 7/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572586.862273:0:112115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788572586.862283:0:112115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788572586.862286:0:112115:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040b:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:0.0:1788572586.862289:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:0.0:1788572586.862291:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.862293:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67240530 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788572586.862296:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:0.0:1788572586.862298:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67240530 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:0.0:1788572586.862304:0:112115:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:0 00000008:00010000:0.0:1788572586.862355:0:112115:0:(osc_cache.c:3059:osc_queue_sync_pages()) ### no dlm lock: extent: ffff92fc58ca8300 00000008:00010000:0.0:1788572586.862387:0:108559:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff92fc58ca8300 00010000:00010000:0.0:1788572586.862406:0:108559:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:0.0:1788572586.862457:0:108559:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff92fc67048000 - object 0x280000400:24 offset 0<>4096 00000100:00100000:0.0:1788572586.862506:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863958656:192.168.202.132@tcp:3:cat.0 00000100:00100000:0.0:1788572586.874577:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863958528:192.168.202.132@tcp:103:ldlm_bl.0 00000008:00010000:1.0:1788572586.876328:0:2393:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff92fc58ca8300 00000020:00200000:1.0:1788572586.876341:0:2393:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1788572586.876360:0:2393:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff92fc58ca8300 00000100:00100000:1.0:1788572586.876388:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863958656:192.168.202.132@tcp:3:cat.0 00000020:00010000:3.0:1788572586.876519:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:3.0:1788572586.876523:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572586.876528:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc67240030 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572586.876538:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572586.876564:0:112115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00010000:3.0:1788572586.876579:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572586.876581:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc67240030 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572586.876590:0:112115:0:(vvp_io.c:533:vvp_io_advance()) advancing 2 bytes 00000080:00200000:3.0:1788572586.876603:0:112115:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572586.876607:0:112115:0:(file.c:2238:ll_file_io_generic()) subfile1: 1 io complete with rc: 0, result: 2, restart: 0 00000080:00200000:3.0:1788572586.876611:0:112115:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 2 00000080:00200008:3.0:1788572586.876653:0:112115:0:(file.c:2483:do_file_read_iter()) START file subfile1:[0x24000040b:0x2:0x0], ppos: 2, count: 4194304 00000080:00200000:3.0:1788572586.876664:0:112115:0:(file.c:2090:ll_file_io_generic()) subfile1: read ppos: 2, bytes: 4194304 00000080:00200000:3.0:1788572586.876668:0:112115:0:(file.c:1848:ll_io_set_mirror()) subfile1: desiginated mirror: 0 00000020:00200000:3.0:1788572586.876674:0:112115:0:(cl_io.c:214:cl_io_rw_init()) header@0000000060999004[0x110, 2, [0x24000040b:0x2:0x0] hash] 00000020:00200000:3.0:1788572586.876676:0:112115:0:(cl_io.c:214:cl_io_rw_init()) io range: 1 [2, 4194306) 0 0 00000080:00200000:3.0:1788572586.876776:0:112115:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572586.876791:0:112115:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 2 chunk: [2, 4194306) 4194306, 4194304 00020000:00200000:3.0:1788572586.876795:0:112115:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572586.876810:0:112115:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 0 [0, 1024] 00000008:00010000:3.0:1788572586.876834:0:112115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:3.0:1788572586.876838:0:112115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc7049dcb0, osc lock ffff92fc4645c130, flags 0x0 00010000:00010000:3.0:1788572586.876857:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.876870:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4198399) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.876882:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.876891:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.876898:0:112115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572586.876907:0:112115:0:(vvp_io.c:835:vvp_io_read_start()) subfile1: read [2, 4194306) 00000001:00010000:3.0:1788572586.876922:0:112115:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x24000040b:0x2:0x0] 00000008:00010000:3.0:1788572586.876927:0:112115:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572586.876929:0:112115:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff92fc7049de30, osc lock ffff92fc4645c390, flags 0x400001002 00010000:00010000:3.0:1788572586.876935:0:112115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.876945:0:112115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:3.0:1788572586.876959:0:112115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572586.876962:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 7/2,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.876969:0:112115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 6/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:3.0:1788572586.876976:0:112115:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572586.876982:0:112115:0:(file.c:1775:ll_merge_attr_nolock()) [0x24000040b:0x2:0x0] updating i_size 2 i_blocks 8 00000020:00010000:3.0:1788572586.876985:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:3.0:1788572586.876988:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572586.876990:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049de30 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572586.876993:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc49de8b90 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572586.876995:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049de30 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572586.877002:0:112115:0:(vvp_io.c:893:vvp_io_read_start()) count:4194304 iocb pos:2 00000020:00010000:3.0:1788572586.877007:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_release():245 00000020:00010000:3.0:1788572586.877010:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572586.877012:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff92fc7049dcb0 (ffff92fc49fceab8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788572586.877016:0:112115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572586.877025:0:112115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff92fc67229000 lock: 00000000b7cec2f0/0x58a604dcef352da8 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xc9a239a2217933b0 expref: -99 pid: 112108 timeout: 0 lvb_type: 1 lru_score: 4 lru_type: 1 00000020:00010000:3.0:1788572586.877032:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc51f28218 (ffff92fc49fceb48/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572586.877035:0:112115:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff92fc7049dcb0 (ffff92fc49fceb48/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572586.877039:0:112115:0:(vvp_io.c:533:vvp_io_advance()) advancing 0 bytes 00000080:00200000:3.0:1788572586.877044:0:112115:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572586.877048:0:112115:0:(file.c:2238:ll_file_io_generic()) subfile1: 1 io complete with rc: 0, result: 0, restart: 0 00000080:00200000:3.0:1788572586.877050:0:112115:0:(file.c:2324:ll_file_io_generic()) iot: 1, result: 0 00000080:00200008:3.0:1788572586.877130:0:112115:0:(file.c:479:ll_file_release()) START file subfile1:[0x24000040b:0x2:0x0](ffff92fc665d0908), flags 100000 00000002:00100000:3.0:1788572586.877186:0:112115:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff92fc4617fb80 x1875449863958400/t4294976780(4294976780) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572586.877197:0:112115:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff92fc67229000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572586.877203:0:112115:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff92fc67339c00 x1875449863958912/t0(0) o35->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:23/10 lens 392/632 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572586.877223:0:112115:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112115:1875449863958912:192.168.202.132@tcp:35:cat.0 00000100:00100000:3.0:1788572586.877304:0:112115:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc603d5a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572586.884751:0:112115:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job cat:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112115:1875449863958912:192.168.202.132@tcp:35:cat.0 00000002:00100000:3.0:1788572586.884786:0:112115:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff92fc4617fb80 x1875449863958400/t4294976780(4294976780) o101->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572602 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788572586.948869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572587.987289:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572588.996936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572590.025156:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572591.052272:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572591.640140:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b62659d6-7f40-4480-a04f-ba5899daf967:2395:1875449863959040:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.640247:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959296:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.640278:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959552:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.640306:0:2395:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572591.640312:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959168:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.640331:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959424:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.659040:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959296:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.659140:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959168:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.659153:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959424:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.659161:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b62659d6-7f40-4480-a04f-ba5899daf967:2395:1875449863959040:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572591.659171:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959552:192.168.202.132@tcp:400:kworker.0 00010000:00010000:1.0:1788572592.072117:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572593.092820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572594.117157:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572595.140891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572596.164911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572596.741624:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b62659d6-7f40-4480-a04f-ba5899daf967:2396:1875449863959680:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572596.741640:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959808:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572596.741757:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863959936:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572596.741786:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863960192:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572596.741939:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863960064:192.168.202.132@tcp:400:kworker.0 00010000:00010000:0.0:1788572597.188897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572598.212903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572599.236889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572600.260877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572601.284882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572601.861008:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b62659d6-7f40-4480-a04f-ba5899daf967:2395:1875449863960320:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572601.861009:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863960448:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572601.861146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863960704:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572601.861157:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863960576:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572601.861200:0:2395:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863960832:192.168.202.132@tcp:400:kworker.0 00010000:00010000:1.0:1788572602.308889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572602.374233:0:112540:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348),name=/ 00000002:00010000:2.0:1788572602.374269:0:112540:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000422:0x1:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572602.374292:0:112540:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572602.374312:0:112540:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572602.374323:0:112540:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000422:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572602.374329:0:112540:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) 00000080:00010000:2.0:1788572602.374333:0:112540:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000422:0x1:0x0](ffff92fc665dd348) for lock 0x58a604dcef352d69 00000080:00010000:2.0:1788572602.374340:0:112540:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x58a604dcef352d69 from it ffff9e0381ce7b58 00010000:00010000:2.0:1788572602.374344:0:112540:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 3/1,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788572602.374355:0:112540:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366800/0x58a604dcef352d69 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xc9a239a221793316 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788572602.374376:0:112540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000422:0x1:0x0](ffff92fc665dd348), inode mode 41ed mask 1 00000080:00200000:2.0:1788572602.374388:0:112540:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d60.sanity-sec, dir=[0x200000422:0x1:0x0](ffff92fc665dd348), flags=8 00000080:00200000:2.0:1788572602.374392:0:112540:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d60.sanity-sec, dir=[0x200000422:0x1:0x0](ffff92fc665dd348), intent=getattr 00000002:00010000:2.0:1788572602.374420:0:112540:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ?y›ø'BfjSÿØâ¼Õ™,[0x0:0x0:0x0]) in obj [0x200000422:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572602.374464:0:112540:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366c00/0x58a604dcef352dbd lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 112540 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572602.374471:0:112540:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366c00/0x58a604dcef352dbd lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112540 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572602.374479:0:112540:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70366c00/0x58a604dcef352dbd lrc: 3/1,0 mode: --/CR res: [0x200000422:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 112540 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572602.374503:0:112540:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617df80 pname:cluuid:pid:xid:nid:opc:job rm:a74e4397-7fb3-41e0-aa97-867f5c09bb42:112540:1875449863960960:192.168.202.132@tcp:101:rm.0 00000100:00100000:2.0:1788572602.374590:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 11 seconds 00010000:00010000:0.0:1788572603.332920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572604.356946:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572605.380922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572606.404964:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572606.981020:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863961216:192.168.202.132@tcp:400:kworker.0 00000100:00100000:0.0:1788572606.981021:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b62659d6-7f40-4480-a04f-ba5899daf967:2394:1875449863961088:192.168.202.132@tcp:400:kworker.0 00000100:00100000:0.0:1788572606.981126:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863961344:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572606.981129:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863961472:192.168.202.132@tcp:400:kworker.0 00000100:00100000:0.0:1788572606.981149:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863961600:192.168.202.132@tcp:400:kworker.0 00010000:00010000:0.0:1788572607.428866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572608.453036:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572609.476905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572610.500962:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572611.524908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572612.100891:0:2395:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff92fc4617e680 x1875449863959808/t0(0) o400->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788572612.100898:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff92fc4617e300 x1875449863959680/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.100951:0:2395:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617e680 x1875449863959808/t0(0) o400->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.100964:0:2395:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788572612.100970:0:2395:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff92fc67229000: Connection to lustre-MDT0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:0.0:1788572612.100977:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b62659d6-7f40-4480-a04f-ba5899daf967:2393:1875449863961728:192.168.202.132@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.101085:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863961984:192.168.202.132@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.101113:0:2393:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863962240:192.168.202.132@tcp:400:kworker.0 00000100:00080000:2.0:1788572612.112892:0:2395:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:1.0:1788572612.112903:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:0:1875449863961856:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572612.112919:0:2394:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863962112:192.168.202.132@tcp:400:kworker.0 00000100:00080000:2.0:1788572612.112923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572612.112928:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572612.112932:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572612.112938:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4341 00000100:00080000:2.0:1788572612.112952:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572612.112962:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572612.112989:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449863962368/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000400:2.0:1788572612.113061:0:2395:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff92fc4617ca80 x1875449863960064/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572612.113080:0:2395:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617ca80 x1875449863960064/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.113086:0:2395:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788572612.113089:0:2395:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff92fc67229000: Connection to lustre-OST0000 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788572612.113093:0:2395:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1788572612.113104:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572612.113106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572612.113108:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572612.113111:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4341 00000100:00080000:2.0:1788572612.113117:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572612.113121:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572612.113130:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449863962496/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572612.113152:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863959808:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572612.113158:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863960064:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572612.113187:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962368:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572612.113223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962496:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572612.115227:0:2396:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617e300 x1875449863959680/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572612.115240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788572612.115244:0:2396:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.132@tcp: Connection to MGS (at 192.168.202.132@tcp) was lost; in progress operations using this service will fail 00000100:00080000:3.0:1788572612.118762:0:2396:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.132@tcp for MGC192.168.202.132@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788572612.118768:0:2396:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788572612.118777:0:2396:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff92fc4617e300 x1875449863959680/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572612.118793:0:2396:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff92fc4617ea00 x1875449863960320/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572617 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572612.118815:0:2396:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff92fc67288000 x1875449863961088/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572622 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572612.118842:0:2396:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff92fc6728bb80 x1875449863961728/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572628 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788572612.118859:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572612.118863:0:2396:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572612.118865:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788572612.118868:0:2396:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:1.0:1788572612.118891:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b62659d6-7f40-4480-a04f-ba5899daf967:2394:1875449863961088:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572612.118896:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b62659d6-7f40-4480-a04f-ba5899daf967:2395:1875449863960320:192.168.202.132@tcp:400:kworker.0 00000100:00000400:3.0:1788572612.118897:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff92fc4617d880 x1875449863959936/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.118914:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572612.118915:0:2396:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617d880 x1875449863959936/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572612.118919:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572612.118922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572612.118924:0:2396:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572612.118928:0:2396:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff92fc67229000: Connection to lustre-MDT0001 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:0.0:1788572612.118931:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b62659d6-7f40-4480-a04f-ba5899daf967:2393:1875449863961728:192.168.202.132@tcp:400:kworker.0 00000100:00080000:2.0:1788572612.118931:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4341 00000100:00080000:3.0:1788572612.118934:0:2396:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1788572612.118947:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00000400:3.0:1788572612.118948:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572596/real 1788572596] req@ffff92fc4617c000 x1875449863960192/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572612.118964:0:2396:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617c000 x1875449863960192/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572612.118971:0:2396:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572612.118975:0:2396:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff92fc67229000: Connection to lustre-OST0001 (at 192.168.202.132@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:2.0:1788572612.118979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863962624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572612.118980:0:2396:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00100000:3.0:1788572612.119002:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b62659d6-7f40-4480-a04f-ba5899daf967:2396:1875449863959680:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572612.119009:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863959936:192.168.202.132@tcp:400:kworker.0 00000100:00080000:2.0:1788572612.119019:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572612.119021:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863960192:192.168.202.132@tcp:400:kworker.0 00000100:00080000:2.0:1788572612.119023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572612.119025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572612.119029:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4341 00000100:00080000:2.0:1788572612.119035:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572612.119041:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863962624:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788572612.119045:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572612.119060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863962752/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572612.119072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572612.119075:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572612.119077:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572612.119081:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4341 00000100:00080000:2.0:1788572612.119087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572612.119092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572612.119103:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449863962880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572612.119123:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572612.119144:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962880:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572612.548966:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788572613.572914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572613.636855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 1 seconds 00010000:00010000:1.0:1788572614.596947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572614.660896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 1 seconds 00010000:00010000:0.0:1788572615.620898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572615.684852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 1 seconds 00010000:00010000:1.0:1788572616.644961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572616.708855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 1 seconds 00000100:00080000:2.0:1788572617.220886:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.220897:0:31488:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572617.220901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.220904:0:31488:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572617.220907:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.220918:0:31488:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572617.220923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.220926:0:31488:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572617.220928:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.220930:0:31488:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:2.0:1788572617.220969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff92fc4617c700 x1875449863960448/t0(0) o400->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788572617.221152:0:2395:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff92fc4617c380 x1875449863960576/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572617.221176:0:2395:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617c380 x1875449863960576/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572617.221182:0:2395:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff92fc5824c800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1788572617.221187:0:2395:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff92fc4617d180 x1875449863960832/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572617.221195:0:2395:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617d180 x1875449863960832/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572617.221199:0:2395:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff92fc58248000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1788572617.221211:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863960576:192.168.202.132@tcp:400:kworker.0 00000100:00100000:3.0:1788572617.221215:0:2395:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2395:1875449863960832:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572617.232935:0:2396:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617c700 x1875449863960448/t0(0) o400->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572617.232950:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff92fc48dc5800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1788572617.232960:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff92fc4617ed80 x1875449863960704/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572617.232974:0:2396:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc4617ed80 x1875449863960704/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572617 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572617.232981:0:2396:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff92fc5824d000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1788572617.232998:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863960448:192.168.202.132@tcp:400:kworker.0 00000100:00100000:2.0:1788572617.233007:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2396:1875449863960704:192.168.202.132@tcp:400:kworker.0 00010000:00010000:0.0:1788572617.668827:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572617.732897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 1 seconds 00000080:00200000:0.0:1788572617.732925:0:110631:0:(vvp_io.c:1826:vvp_io_init()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572617.732944:0:110631:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572617.732978:0:110631:0:(vvp_io.c:304:vvp_io_fini()) [0x200000422:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572617.732986:0:110631:0:(vvp_io.c:1826:vvp_io_init()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572617.732991:0:110631:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572617.732997:0:110631:0:(vvp_io.c:304:vvp_io_fini()) [0x24000040b:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1788572618.692938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572618.756862:0:112540:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572602/real 1788572602] req@ffff92fc4617df80 x1875449863960960/t0(0) o101->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 592/1376 e 0 to 1 dl 1788572618 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572618.765683:0:112540:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff92fc48dc5800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1788572618.765692:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572619.716829:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572619.780839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572620.740925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572620.804846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572621.764847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572621.828848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572622.340847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.340861:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:1.0:1788572622.340863:0:2393:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff92fc67288e00 x1875449863961216/t0(0) o400->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.340863:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc4617e680 x1875449863962752/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572622.340866:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.340875:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572622.340880:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.340887:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572622.340893:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.340898:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572622.340902:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572622.340905:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572622.340918:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449863962752/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572622.340936:0:2394:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff92fc67289500 x1875449863961344/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.340937:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc4617f480 x1875449863962880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.340976:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449863962880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.340991:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc4617ca80 x1875449863962624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572622.340992:0:2394:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc67289500 x1875449863961344/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.341005:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449863962624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572622.341035:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff92fc5824c800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1788572622.341035:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc4617dc00 x1875449863962368/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572622.341050:0:2394:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff92fc67289c00 x1875449863961600/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.341054:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449863962368/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.341086:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc4617f800 x1875449863962496/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572622.341092:0:2394:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc67289c00 x1875449863961600/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572622.341111:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449863962496/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572622.341127:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff92fc58248000 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:2.0:1788572622.341138:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788572622.341151:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863961344:192.168.202.132@tcp:400:kworker.0 00000100:00100000:0.0:1788572622.341160:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863961600:192.168.202.132@tcp:400:kworker.0 00000100:00080000:2.0:1788572622.341165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572622.341172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962752:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788572622.341181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572622.341192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788572622.341194:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572622.341198:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962880:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788572622.341201:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572622.341207:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572622.341210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572622.341211:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572622.341216:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572622.341219:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572622.341222:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572622.341227:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4367 00000100:00080000:2.0:1788572622.341232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572622.341239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863962624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572622.341239:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572622.341250:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788572622.341251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572622.341263:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788572622.341266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572622.341274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962368:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788572622.341279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572622.341288:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572622.341290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863962496:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572622.341297:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449863963008/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572622.341309:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572622.341312:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572622.341314:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572622.341318:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4367 00000100:00080000:3.0:1788572622.341322:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572622.341325:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572622.341329:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572622.341330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963008:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572622.341339:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863963136/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572622.341348:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572622.341350:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572622.341352:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572622.341354:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4367 00000100:00080000:3.0:1788572622.341357:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572622.341360:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572622.341363:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572622.341373:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449863963264/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572622.341382:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572622.341384:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572622.341386:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572622.341388:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4367 00000100:00080000:3.0:1788572622.341391:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572622.341394:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572622.341404:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449863963392/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572622.341410:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572622.341413:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572622.341414:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572622.341416:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4367 00000100:00080000:3.0:1788572622.341418:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572622.341420:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572622.341423:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572622.341433:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863963520/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572622.341444:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963136:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572622.341494:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963264:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572622.341535:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863963392:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572622.341562:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572622.350939:0:2393:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc67288e00 x1875449863961216/t0(0) o400->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572622.350953:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff92fc48dc5800 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:1.0:1788572622.350964:0:2393:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572606/real 1788572606] req@ffff92fc6728b800 x1875449863961472/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572622.350978:0:2393:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc6728b800 x1875449863961472/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572622 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572622.350985:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff92fc5824d000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1788572622.351002:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863961216:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572622.351009:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863961472:192.168.202.132@tcp:400:kworker.0 00010000:00010000:1.0:1788572622.788949:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572622.852895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572623.812861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572623.876862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572624.836854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572624.900857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572625.860879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572625.924876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572626.884907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572626.948831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572627.460895:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.460906:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.460909:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.460912:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.460914:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.460966:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.460968:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.460970:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.460972:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.460974:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1788572627.908851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572627.972854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00000400:1.0:1788572628.484836:0:2393:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc6728a680 x1875449863962240/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788572628.484931:0:2394:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc67288380 x1875449863962112/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572628.484955:0:2394:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc67288380 x1875449863962112/t0(0) o400->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572628.484962:0:2394:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff92fc5824d000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1788572628.484971:0:2394:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2394:1875449863962112:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572628.500834:0:2393:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc6728a680 x1875449863962240/t0(0) o400->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572628.500846:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff92fc58248000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:1.0:1788572628.500857:0:2393:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff92fc67289f80 x1875449863961984/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.500873:0:2393:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff92fc67289f80 x1875449863961984/t0(0) o400->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572628.500880:0:2393:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff92fc5824c800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1788572628.500896:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863962240:192.168.202.132@tcp:400:kworker.0 00000100:00100000:1.0:1788572628.500904:0:2393:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2393:1875449863961984:192.168.202.132@tcp:400:kworker.0 00010000:00010000:1.0:1788572628.932898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572628.996828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572629.956876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572630.020814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572630.980936:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572631.044823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572632.004862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572632.068815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572632.580839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.580851:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.580855:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.580858:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.580861:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.580863:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.580865:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.580869:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.580872:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.580875:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788572633.028888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572633.092828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572634.052855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572634.116857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572635.076874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572635.140834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000800:00000100:0.0:1788572635.332857:0:2384:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.202.132@tcp (192.168.202.132:988) timed out; the network or node may be down. 00010000:00010000:0.0:1788572636.100864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572636.164799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572637.124902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572637.188871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572637.700811:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.700820:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572637.700822:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff92fc6733bb80 x1875449863963520/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572637.700823:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.700831:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572637.700843:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.700846:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572637.700849:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.700852:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572637.700855:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863963520/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572637.700857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.700865:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572637.700877:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff92fc67339c00 x1875449863963136/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700888:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449863963136/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700897:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff92fc67339180 x1875449863963264/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700909:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449863963264/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700919:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff92fc67339f80 x1875449863963392/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700932:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449863963392/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700951:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff92fc6733ad80 x1875449863963008/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572637.700961:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449863963008/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572637 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572637.700976:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572637.700989:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572637.700992:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963520:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788572637.700996:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572637.701000:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572637.701025:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963136:192.168.202.132@tcp:8:kworker.0 00000100:00080000:2.0:1788572637.701029:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572637.701034:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572637.701037:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963264:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788572637.701041:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.701046:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572637.701049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572637.701050:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572637.701057:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572637.701061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572637.701064:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572637.701071:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863963392:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572637.701072:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4378 00000100:00080000:2.0:1788572637.701076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572637.701081:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572637.701084:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963008:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788572637.701086:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572637.701096:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572637.701107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572637.701136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863963648/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572637.701155:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.701158:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572637.701159:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572637.701162:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4378 00000100:00080000:3.0:1788572637.701165:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572637.701168:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572637.701172:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572637.701181:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863963776/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572637.701183:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963648:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788572637.701202:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.701204:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572637.701206:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572637.701210:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4378 00000100:00080000:3.0:1788572637.701212:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572637.701214:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572637.701218:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572637.701227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863963904/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572637.701234:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.701236:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572637.701237:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572637.701239:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4378 00000100:00080000:3.0:1788572637.701242:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572637.701245:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572637.701248:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572637.701254:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863964032/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572637.701263:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572637.701266:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572637.701267:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572637.701270:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4378 00000100:00080000:3.0:1788572637.701274:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572637.701276:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572637.701287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963776:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572637.701289:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863964160/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572637.701323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963904:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572637.701338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964032:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572637.701356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863964160:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572638.148873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572638.212852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572639.172872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572639.236857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572640.196837:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572640.260813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572641.220826:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572641.284790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572642.244853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572642.308830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572642.820791:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.820801:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.820805:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.820808:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.820810:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.820813:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.820815:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.820817:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572642.820820:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572642.820822:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788572643.268986:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572643.332829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572644.292885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572644.356895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572645.316917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572645.380803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572646.340895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572646.404861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572647.364904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572647.428808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572647.876925:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.132:988 00000400:00000100:0.0:1788572647.876973:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572647.876992:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572647.877025:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572647.877031:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572647.877038:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572647.877044:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788572647.877068:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572647] req@ffff92fc67339880 x1875449863964160/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877089:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449863964160/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877098:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572647.877115:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877118:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863964160:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572647.877128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572647] req@ffff92fc67338a80 x1875449863963776/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863963776/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572647.877157:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572647.877162:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572647.877165:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572647.877166:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963776:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788572647.877176:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4393 00000100:00100000:2.0:1788572647.877181:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572647] req@ffff92fc6733b100 x1875449863963904/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863963904/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572647.877197:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1788572647.877198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572647.877210:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788572647.877217:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877220:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963904:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572647.877228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572647] req@ffff92fc67338380 x1875449863964032/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572647.877245:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863964288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572647.877246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863964032/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572647.877278:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572647.877284:0:110631:0:(pinger.c:247:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572647.877287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788572647.877290:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572647.877293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964032:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572647.877304:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572637/real 1788572647] req@ffff92fc6733bb80 x1875449863963648/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572647.877306:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572647.877311:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572647.877321:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4393 00000100:00080000:3.0:1788572647.877329:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 15s 00000100:00100000:2.0:1788572647.877334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863963648/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572647.877338:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788572647.877351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572647.877355:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788572647.877365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863963648:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572647.877375:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863964416/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572647.877389:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572647.877393:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572647.877395:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572647.877401:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4393 00000100:00080000:3.0:1788572647.877406:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572647.877409:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572647.877414:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572647.877423:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863964288:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572647.877424:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863964544/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572647.877437:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572647.877439:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572647.877441:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572647.877443:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4393 00000100:00080000:3.0:1788572647.877446:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572647.877449:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572647.877454:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572647.877466:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863964672/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572647.877481:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572647.877484:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572647.877486:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788572647.877491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572647.877493:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4393 00000100:00080000:3.0:1788572647.877500:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572647.877505:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572647.877510:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572647.877519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572647] req@ffff92fc67338000 x1875449863964288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572647.877526:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863964800/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572647.877534:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863964288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572647.877547:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877549:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863964288:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572647.877555:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964416:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572647.877571:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572647.877580:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572647] req@ffff92fc67339500 x1875449863964416/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877587:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863964416/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877597:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572647.877599:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877601:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964416:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572647.877610:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964544:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572647.877626:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572647.877636:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572647] req@ffff92fc6733b480 x1875449863964544/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877648:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863964544/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877657:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572647.877660:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877664:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964544:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572647.877671:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964672:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572647.877757:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572647.877774:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572647] req@ffff92fc6733a300 x1875449863964672/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877797:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863964672/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877806:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572647.877808:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877812:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964672:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572647.877816:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572647.877836:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572647.877847:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572647/real 1788572647] req@ffff92fc67338700 x1875449863964800/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572647.877860:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863964800/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572647.877870:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572647.877873:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572647.877876:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863964800:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788572648.388864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572648.452814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572649.412968:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572649.476849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572650.436893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572650.500888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572651.460932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572651.524797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572652.484913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572652.548877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572653.060875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572653.060887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572653.060892:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572653.060904:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4403 00000100:00080000:3.0:1788572653.060916:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572653.060925:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572653.060995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863964928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572653.061030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572653.061035:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572653.061038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572653.061042:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4403 00000100:00080000:3.0:1788572653.061047:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572653.061051:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572653.061065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572653.061083:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449863965056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572653.061098:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863964928:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572653.061104:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572653.061110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572653.061113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572653.061118:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4403 00000100:00080000:3.0:1788572653.061123:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572653.061128:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572653.061135:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572653.061152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449863965184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572653.061163:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572653.061167:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572653.061170:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572653.061175:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4403 00000100:00080000:3.0:1788572653.061179:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572653.061183:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572653.061189:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572653.061201:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449863965312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572653.061209:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572653.061211:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572653.061213:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572653.061216:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4403 00000100:00080000:3.0:1788572653.061219:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572653.061222:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572653.061225:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572653.061233:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863965440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572653.061270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965056:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572653.061312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965184:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572653.061339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965312:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572653.061368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965440:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572653.508958:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572653.572850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572654.532824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572654.596844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572655.557821:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572655.620890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572656.132965:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.132:988 00000400:00000100:0.0:1788572656.133000:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572656.133015:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572656.133050:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572656.133061:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572656.133071:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572656.133080:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788572656.133101:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572653/real 1788572656] req@ffff92fc6733aa00 x1875449863965056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572656.133128:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449863965056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572656.133140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572656.133145:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572656.133150:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965056:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572656.133157:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572653/real 1788572656] req@ffff92fc67338e00 x1875449863965184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572656.133169:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449863965184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572656.133177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572656.133180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572656.133183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965184:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572656.133190:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572653/real 1788572656] req@ffff92fc6733a680 x1875449863965312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572656.133200:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449863965312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572656.133207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572656.133210:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572656.133213:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965312:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572656.133220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572653/real 1788572656] req@ffff92fc67338700 x1875449863965440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572656.133229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863965440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572656.133235:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572656.133238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572656.133241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965440:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572656.133247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572653/real 1788572656] req@ffff92fc6733b800 x1875449863964928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572656.133258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863964928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572656.133264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572656.133268:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572656.133271:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863964928:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572656.580820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572656.644852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572657.604875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572657.669187:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572658.180908:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572658.180920:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572658.180925:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572658.180937:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4408 00000100:00080000:0.0:1788572658.180949:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572658.180958:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572658.181002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863965568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572658.181036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572658.181040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572658.181042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572658.181046:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4408 00000100:00080000:0.0:1788572658.181050:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572658.181055:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572658.181068:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572658.181082:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863965568:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572658.181083:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863965696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572658.181099:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572658.181102:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572658.181105:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572658.181109:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4408 00000100:00080000:0.0:1788572658.181113:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572658.181117:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572658.181133:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572658.181148:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449863965824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572658.181159:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572658.181163:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572658.181165:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572658.181169:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4408 00000100:00080000:0.0:1788572658.181174:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572658.181178:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572658.181185:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572658.181198:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449863965952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572658.181207:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572658.181211:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572658.181213:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572658.181216:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4408 00000100:00080000:0.0:1788572658.181221:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572658.181226:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572658.181231:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572658.181241:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965696:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572658.181246:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449863966080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572658.181280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965824:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572658.181308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965952:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572658.181339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966080:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788572658.629385:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572658.693119:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788572659.205086:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.132:988 00000400:00000100:3.0:1788572659.205132:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572659.205146:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572659.205180:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572659.205189:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572659.205197:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572659.205203:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788572659.205270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572659] req@ffff92fc67049880 x1875449863966080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572659.205295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449863966080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572659.205308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572659.205313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572659.205317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966080:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572659.205326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572659] req@ffff92fc6704a300 x1875449863965696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572659.205344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449863965696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572659.205365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572659.205370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572659.205373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965696:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572659.205381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572659] req@ffff92fc6704aa00 x1875449863965824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572659.205392:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449863965824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572659.205400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572659.205404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572659.205407:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965824:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572659.205414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572659] req@ffff92fc67049f80 x1875449863965952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572659.205426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449863965952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572659.205433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572659.205436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572659.205439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863965952:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572659.205446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572658/real 1788572659] req@ffff92fc6704a680 x1875449863965568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572659.205458:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449863965568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572659.205465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572659.205469:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572659.205472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863965568:192.168.202.132@tcp:250:kworker.0 00010000:00010000:1.0:1788572659.652852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572659.716843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788572660.676941:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572660.741007:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572661.700915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572661.764897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788572662.724889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572662.788775:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572663.300858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572663.300871:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572663.300876:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572663.300886:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4413 00000100:00080000:1.0:1788572663.300897:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572663.300907:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572663.300954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449863966208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572663.300981:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572663.300985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572663.300986:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572663.300989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4413 00000100:00080000:1.0:1788572663.300992:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572663.300994:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572663.301004:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572663.301013:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449863966336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572663.301022:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572663.301024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572663.301026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572663.301032:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4413 00000100:00080000:1.0:1788572663.301035:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572663.301035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863966208:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572663.301045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572663.301052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572663.301085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449863966464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572663.301092:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572663.301095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572663.301096:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572663.301100:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4413 00000100:00080000:1.0:1788572663.301103:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572663.301106:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572663.301110:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572663.301125:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449863966592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572663.301132:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572663.301134:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572663.301136:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572663.301139:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966336:192.168.202.132@tcp:38:kworker.0 00000100:00080000:1.0:1788572663.301143:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4413 00000100:00080000:1.0:1788572663.301156:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572663.301162:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572663.301168:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572663.301177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966464:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572663.301186:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449863966720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572663.301200:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966592:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572663.301236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966720:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572663.748948:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572663.813018:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788572664.772874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572664.836865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572665.796866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572665.860915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572665.861095:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.132:988 00000400:00000100:0.0:1788572665.861136:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572665.861167:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572665.861267:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572665.861279:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572665.861289:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572665.861301:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788572665.861308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572665] req@ffff92fc6728a680 x1875449863966336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572665.861343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449863966336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572665.861358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572665.861364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572665.861368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966336:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572665.861376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572665] req@ffff92fc6728b800 x1875449863966464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572665.861388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449863966464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572665.861397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572665.861401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572665.861404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966464:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572665.861428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572665] req@ffff92fc67288e00 x1875449863966592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572665.861441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449863966592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572665.861450:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572665.861455:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572665.861458:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966592:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572665.861464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572665] req@ffff92fc67289f80 x1875449863966208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572665.861477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449863966208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572665.861486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572665.861490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572665.861494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863966208:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572665.861529:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572663/real 1788572665] req@ffff92fc67288000 x1875449863966720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572665.861541:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449863966720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572665.861548:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572665.861552:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572665.861555:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966720:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788572666.820883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572666.884864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572667.844857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572667.908885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572668.420826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572668.420836:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572668.420841:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572668.420851:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4419 00000100:00080000:1.0:1788572668.420862:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572668.420869:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572668.420903:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449863966848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572668.420934:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572668.420939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572668.420941:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572668.420944:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4419 00000100:00080000:1.0:1788572668.420948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572668.420952:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572668.420962:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572668.420976:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449863966976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572668.420985:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863966848:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572668.421005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572668.421009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572668.421012:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572668.421016:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4419 00000100:00080000:1.0:1788572668.421022:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572668.421026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572668.421032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572668.421047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449863967104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572668.421057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788572668.421060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572668.421075:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572668.421077:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572668.421088:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4419 00000100:00080000:1.0:1788572668.421097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572668.421101:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572668.421106:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572668.421119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff92fc67289880 x1875449863966848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572668.421120:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449863967232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572668.421135:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572668.421138:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572668.421140:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572668.421144:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4419 00000100:00080000:1.0:1788572668.421147:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572668.421150:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572668.421154:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572668.421162:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449863967360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572668.421216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449863966848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.421227:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.421233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572668.421237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863966848:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572668.421259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966976:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572668.421291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572668.421310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff92fc6728aa00 x1875449863966976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.421324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449863966976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.421333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.421338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572668.421342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863966976:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572668.421351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967104:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572668.421376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572668.421388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff92fc6728b480 x1875449863967104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.421400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449863967104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.421407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.421411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572668.421414:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967104:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572668.421538:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967232:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572668.421563:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572668.421654:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff92fc6728ad80 x1875449863967232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.421670:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449863967232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.421690:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.421694:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572668.421698:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967232:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572668.421704:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967360:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572668.421750:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572668.421767:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572668] req@ffff92fc6728a300 x1875449863967360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572668.421789:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449863967360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572668.421798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572668.421801:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572668.421805:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967360:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788572668.868962:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572668.932865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572669.892944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572669.956873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788572670.916986:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572670.980814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572671.940880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572672.004852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788572672.964950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572673.028870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572673.540837:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572673.540846:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572673.540850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572673.540858:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4424 00000100:00080000:2.0:1788572673.540867:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572673.540873:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572673.540921:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449863967488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572673.540950:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572673.540954:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572673.540956:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572673.540961:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4424 00000100:00080000:2.0:1788572673.540964:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572673.540968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572673.540977:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572673.540990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449863967616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572673.540992:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863967488:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572673.541001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572673.541004:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572673.541006:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572673.541010:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4424 00000100:00080000:2.0:1788572673.541013:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572673.541017:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572673.541021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572673.541030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863967744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572673.541038:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572673.541041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572673.541042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572673.541045:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4424 00000100:00080000:2.0:1788572673.541048:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572673.541052:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572673.541056:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572673.541064:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449863967872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572673.541072:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572673.541075:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572673.541076:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572673.541080:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4424 00000100:00080000:2.0:1788572673.541083:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572673.541086:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572673.541090:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572673.541097:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572673.541098:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863968000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572673.541128:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572673.541154:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572673.541175:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968000:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572673.988887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572674.052830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572675.012907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572675.076844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572676.036876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572676.100823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572676.613089:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.132:988 00000400:00000100:0.0:1788572676.613146:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572676.613164:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572676.613211:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572676.613220:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572676.613228:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572676.613238:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788572676.613277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff92fc4617e680 x1875449863968000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.613312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449863968000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.613329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.613336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572676.613342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968000:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572676.613352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff92fc4617dc00 x1875449863967616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.613376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449863967616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.613385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.613390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572676.613394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572676.613403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff92fc4617ca80 x1875449863967744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.613418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449863967744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.613426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.613430:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572676.613433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572676.613441:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff92fc4617f480 x1875449863967872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.613452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449863967872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.613460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.613465:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572676.613468:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863967872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572676.613477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572673/real 1788572676] req@ffff92fc4617f800 x1875449863967488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572676.613492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449863967488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572676.613501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572676.613505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572676.613509:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863967488:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572677.060903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572677.124856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572678.084849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572678.148862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572678.660903:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572678.660914:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572678.660919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572678.660931:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4429 00000100:00080000:1.0:1788572678.660944:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572678.660952:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572678.660996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449863968128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572678.661034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572678.661042:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572678.661044:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572678.661049:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4429 00000100:00080000:1.0:1788572678.661053:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572678.661057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572678.661071:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572678.661088:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449863968256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572678.661102:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572678.661106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572678.661111:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572678.661112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863968128:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572678.661118:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4429 00000100:00080000:1.0:1788572678.661125:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572678.661128:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572678.661133:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572678.661146:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449863968384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572678.661154:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572678.661159:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572678.661161:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572678.661166:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4429 00000100:00080000:1.0:1788572678.661169:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572678.661173:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572678.661180:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572678.661195:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449863968512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572678.661205:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572678.661210:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572678.661212:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572678.661216:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4429 00000800:00000100:3.0:1788572678.661233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788572678.661234:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572678.661244:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572678.661250:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572678.661266:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449863968640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572678.661314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff92fc6728a300 x1875449863968128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.661340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449863968128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.661352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.661357:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572678.661362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863968128:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572678.661397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968256:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572678.661427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572678.661443:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff92fc6728ad80 x1875449863968256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.661456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449863968256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.661465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.661470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572678.661473:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968256:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572678.661481:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968384:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572678.661503:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572678.661517:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff92fc6728b480 x1875449863968384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.661528:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449863968384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.661536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.661539:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572678.661543:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968384:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572678.661549:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968512:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572678.661572:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572678.661584:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff92fc6728aa00 x1875449863968512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.661597:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449863968512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.661787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.661793:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572678.661798:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968512:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572678.661806:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968640:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572678.661844:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572678.661861:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572678] req@ffff92fc67289880 x1875449863968640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572678.661876:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449863968640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572678.661886:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572678.661890:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572678.661894:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968640:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572679.108876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572679.172891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572680.132885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572680.196830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572681.156888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572681.220867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572682.180927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572682.244839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572683.204889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572683.268867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572683.780866:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.780875:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572683.780879:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.780896:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4434 00000100:00080000:3.0:1788572683.780906:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.780912:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572683.780946:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863968768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572683.780975:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.780978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572683.780979:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.780982:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4434 00000100:00080000:3.0:1788572683.780984:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.780987:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572683.780995:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572683.781004:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863968896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572683.781012:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.781014:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572683.781016:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.781018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4434 00000100:00080000:3.0:1788572683.781020:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.781023:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572683.781026:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572683.781032:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863969024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572683.781044:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863968768:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572683.781046:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.781062:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572683.781066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.781069:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4434 00000100:00080000:3.0:1788572683.781071:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.781073:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572683.781078:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572683.781087:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863969152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572683.781100:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.781102:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572683.781104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.781106:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4434 00000100:00080000:3.0:1788572683.781109:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.781111:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572683.781116:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572683.781125:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863969280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572683.781148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572683.781198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff92fc6733a300 x1875449863968768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.781223:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863968768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.781236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.781241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572683.781245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863968768:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572683.781251:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968896:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572683.781273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572683.781283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff92fc6733b480 x1875449863968896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.781292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863968896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.781298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.781301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572683.781303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863968896:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572683.781330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969024:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572683.781342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572683.781349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff92fc67339500 x1875449863969024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.781362:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863969024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.781367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.781369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572683.781371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969024:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572683.781375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969152:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572683.781387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572683.781393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff92fc67338000 x1875449863969152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.781401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863969152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.781406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.781408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572683.781409:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969152:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572683.781425:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969280:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572683.781437:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572683.781444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff92fc6733bb80 x1875449863969280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.781452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863969280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.781462:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.781464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572683.781466:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969280:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572684.228947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572684.292979:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572685.252908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572685.316906:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572686.276860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572686.340989:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572687.300894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572687.364867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572688.324881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572688.388797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572688.900842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.900854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572688.900858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.900869:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4439 00000100:00080000:3.0:1788572688.900882:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.900890:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572688.900934:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863969408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.900966:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.900970:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572688.900971:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.900975:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4439 00000100:00080000:3.0:1788572688.900977:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.900986:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572688.900997:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572688.901009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863969536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.901021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572688.901027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863969408:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572688.901027:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572688.901030:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.901038:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4439 00000100:00080000:3.0:1788572688.901041:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.901043:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572688.901047:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572688.901056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863969664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.901063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.901065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572688.901067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.901070:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4439 00000100:00080000:3.0:1788572688.901089:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.901094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572688.901098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572688.901105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572688.901113:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863969792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.901125:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.901129:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572688.901131:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.901135:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4439 00000100:00080000:3.0:1788572688.901140:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.901144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572688.901152:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572688.901155:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff92fc67338380 x1875449863969408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572688.901168:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449863969920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572688.901174:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863969408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.901187:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.901192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572688.901195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863969408:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572688.901222:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969536:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572688.901238:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572688.901247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff92fc6733b100 x1875449863969536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.901256:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863969536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.901262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.901264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572688.901266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969536:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572688.901271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969664:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572688.901282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572688.901289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff92fc67338a80 x1875449863969664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.901297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863969664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.901302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.901303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572688.901305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969664:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572688.901309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969792:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572688.901323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572688.901329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff92fc67339880 x1875449863969792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.901337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449863969792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.901343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.901345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572688.901347:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969792:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572688.901350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969920:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572688.901361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572688.901369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572688] req@ffff92fc6733ad80 x1875449863969920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572688.901383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449863969920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572688.901392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572688.901396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572688.901399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863969920:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572689.348893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572689.412849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572690.372893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572690.436783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572691.396914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572691.460855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572692.420945:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572692.484798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572693.444888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572693.508792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572694.020808:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572694.020817:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572694.020820:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572694.020829:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4444 00000100:00080000:3.0:1788572694.020839:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572694.020847:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572694.020888:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449863970048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572694.020913:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572694.020916:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572694.020918:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572694.020921:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4444 00000100:00080000:3.0:1788572694.020924:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572694.020926:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572694.020934:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572694.020943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449863970176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572694.020951:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572694.020953:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572694.020955:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572694.020958:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4444 00000100:00080000:3.0:1788572694.020961:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572694.020963:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572694.020966:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572694.020973:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863970048:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572694.020975:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863970304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572694.020987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572694.020988:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572694.020990:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572694.020992:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4444 00000100:00080000:3.0:1788572694.020994:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572694.020996:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572694.020999:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572694.021009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449863970432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572694.021015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572694.021018:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572694.021019:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572694.021022:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4444 00000100:00080000:3.0:1788572694.021031:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572694.021034:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572694.021037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572694.021045:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863970560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572694.021077:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970176:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572694.021122:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970304:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572694.021144:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970432:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572694.021158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970560:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572694.468987:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572694.532877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572695.492851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572695.556839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572696.516920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572696.580831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572697.093005:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.132:988 00000400:00000100:0.0:1788572697.093048:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572697.093066:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572697.093111:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572697.093123:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572697.093134:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572697.093144:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788572697.093152:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572694/real 1788572697] req@ffff92fc67339c00 x1875449863970304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572697.093180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449863970304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572697.093193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572697.093198:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572697.093202:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970304:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572697.093211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572694/real 1788572697] req@ffff92fc6733ad80 x1875449863970432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572697.093231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449863970432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572697.093240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572697.093243:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572697.093247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970432:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572697.093265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572694/real 1788572697] req@ffff92fc67339880 x1875449863970560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572697.093289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449863970560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572697.093301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572697.093305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572697.093308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970560:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572697.093315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572694/real 1788572697] req@ffff92fc67339f80 x1875449863970048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572697.093327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449863970048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572697.093334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572697.093338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572697.093341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863970048:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572697.093347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572694/real 1788572697] req@ffff92fc67339180 x1875449863970176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572697.093359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449863970176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572697.093368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572697.093371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572697.093374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970176:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788572697.540831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572697.604837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572698.564901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572698.628841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572699.140825:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572699.140834:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572699.140839:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572699.140849:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4449 00000100:00080000:1.0:1788572699.140860:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572699.140874:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572699.140906:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449863970688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572699.140927:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572699.140930:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572699.140932:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572699.140937:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4449 00000100:00080000:1.0:1788572699.140941:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572699.140947:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572699.140956:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863970688:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788572699.140957:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572699.140974:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449863970816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572699.140987:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572699.140991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572699.140993:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572699.141003:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4449 00000100:00080000:1.0:1788572699.141008:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572699.141013:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572699.141019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572699.141028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572699.141036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449863970944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572699.141048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572699.141051:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572699.141053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572699.141057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4449 00000100:00080000:1.0:1788572699.141060:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572699.141065:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572699.141070:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572699.141084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449863971072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572699.141090:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572699/real 1788572699] req@ffff92fc67288000 x1875449863970688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572699.141099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572699.141110:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572699.141114:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572699.141119:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4449 00000100:00080000:1.0:1788572699.141124:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572699.141129:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449863970688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572699.141132:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572699.141138:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572699.141157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788572699.141158:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449863971200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572699.141168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572699.141178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863970688:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572699.141213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970816:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572699.141258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572699.141276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572699/real 1788572699] req@ffff92fc67289f80 x1875449863970816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572699.141291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449863970816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572699.141307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572699.141311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572699.141316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970816:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572699.141323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970944:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572699.141347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572699.141361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572699/real 1788572699] req@ffff92fc67288e00 x1875449863970944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572699.141375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449863970944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572699.141384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572699.141388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572699.141391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863970944:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572699.141398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971072:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572699.141421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572699.141434:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572699/real 1788572699] req@ffff92fc6728b800 x1875449863971072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572699.141446:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449863971072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572699.141454:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572699.141457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572699.141461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971072:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572699.141467:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971200:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572699.141486:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572699.141497:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572699/real 1788572699] req@ffff92fc6728a680 x1875449863971200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572699.141509:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449863971200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572699.141517:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572699.141521:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572699.141524:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971200:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572699.588959:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572699.652852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572700.612933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572700.676842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572701.636875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572701.700821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572702.660907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572702.724837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572703.684928:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572703.748848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572704.260830:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572704.260845:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572704.260851:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572704.260860:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4454 00000100:00080000:0.0:1788572704.260869:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572704.260876:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572704.260907:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863971328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572704.260931:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572704.260934:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572704.260936:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572704.260940:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4454 00000100:00080000:0.0:1788572704.260943:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572704.260946:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572704.260955:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572704.260968:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449863971456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572704.260978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572704.260980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572704.260982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572704.260985:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4454 00000100:00080000:0.0:1788572704.260988:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572704.260992:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572704.260996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572704.261001:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863971328:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572704.261008:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449863971584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572704.261017:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572704.261021:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572704.261023:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572704.261028:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4454 00000100:00080000:0.0:1788572704.261032:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572704.261037:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572704.261042:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572704.261056:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449863971712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572704.261067:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572704.261071:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572704.261073:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572704.261078:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4454 00000100:00080000:0.0:1788572704.261083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572704.261088:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572704.261105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572704.261118:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449863971840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572704.261145:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572704.261300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff92fc6704ad80 x1875449863971328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.261318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449863971328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.261327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.261330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572704.261378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863971328:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572704.261388:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971456:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572704.261414:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572704.261427:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff92fc6704b100 x1875449863971456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.261436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449863971456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.261442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.261445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572704.261447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971456:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572704.261465:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971584:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572704.261478:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572704.261486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff92fc6704b480 x1875449863971584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.261494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449863971584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.261499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.261501:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572704.261503:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971584:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572704.261506:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971712:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572704.261516:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572704.261522:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff92fc6704b800 x1875449863971712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.261529:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449863971712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.261534:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.261536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572704.261538:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971712:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572704.261541:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971840:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572704.261551:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572704.261557:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572704/real 1788572704] req@ffff92fc67049c00 x1875449863971840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572704.261564:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449863971840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572704.261569:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572704.261571:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572704.261573:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863971840:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572704.708901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572704.772920:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572705.732873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572705.796868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572706.756903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572706.820911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572707.780879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572707.844799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572708.804902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572708.868836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572709.380829:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572709.380838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572709.380842:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572709.380850:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4459 00000100:00080000:0.0:1788572709.380860:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572709.380865:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572709.380898:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449863971968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572709.380921:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572709.380932:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572709.380934:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572709.380939:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4459 00000100:00080000:0.0:1788572709.380943:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572709.380945:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572709.380954:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572709.380979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449863972096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572709.380982:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863971968:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572709.381009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572709.381012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572709.381015:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572709.381018:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4459 00000100:00080000:0.0:1788572709.381022:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572709.381025:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572709.381028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572709.381040:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449863972224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572709.381047:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572709.381049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572709.381050:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572709.381053:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4459 00000100:00080000:0.0:1788572709.381055:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572709.381059:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572709.381061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572709.381067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449863972352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572709.381074:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572709.381076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572709.381077:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572709.381080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4459 00000100:00080000:0.0:1788572709.381083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572709.381085:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572709.381088:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572709.381095:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449863972480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572709.381104:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572709.381160:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff92fc67049500 x1875449863971968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.381180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449863971968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.381189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.381193:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572709.381196:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863971968:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572709.381221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972096:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572709.381240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572709.381250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff92fc67048e00 x1875449863972096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.381258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449863972096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.381264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.381266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572709.381268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972096:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572709.381274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972224:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572709.381286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572709.381293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff92fc6704bb80 x1875449863972224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.381302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449863972224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.381307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.381309:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572709.381311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972224:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572709.381315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972352:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572709.381327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572709.381334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff92fc67049180 x1875449863972352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.381341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449863972352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.381347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.381349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572709.381351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972352:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572709.381355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972480:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572709.381366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572709.381372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572709/real 1788572709] req@ffff92fc67049c00 x1875449863972480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.381380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449863972480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.381385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.381387:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572709.381389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972480:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572709.828832:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572709.892822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572710.852954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572710.916869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572711.876890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572711.940843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572712.900875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572712.964842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572713.924884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572713.988894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572714.500856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.500868:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572714.500873:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.500884:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4465 00000100:00080000:0.0:1788572714.500895:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.500904:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572714.500942:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449863972608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572714.500968:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.500973:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572714.500975:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.500980:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4465 00000100:00080000:0.0:1788572714.500984:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.500989:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572714.500994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863972608:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788572714.501002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572714.501022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449863972736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572714.501035:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.501039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572714.501042:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.501046:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4465 00000100:00080000:0.0:1788572714.501051:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572714.501054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788572714.501059:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572714.501065:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572714.501077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449863972864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572714.501087:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.501092:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788572714.501094:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff92fc6704b800 x1875449863972608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572714.501095:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572714.501105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4465 00000100:00080000:0.0:1788572714.501111:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.501115:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572714.501118:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449863972608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788572714.501121:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788572714.501132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788572714.501137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863972992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572714.501140:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788572714.501149:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572714.501151:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572714.501153:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572714.501156:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863972608:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572714.501159:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4465 00000100:00080000:0.0:1788572714.501164:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572714.501167:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572714.501170:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572714.501177:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863973120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572714.501182:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972736:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572714.501207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572714.501219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff92fc6704b480 x1875449863972736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.501227:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449863972736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.501237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.501240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572714.501242:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972736:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572714.501246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972864:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572714.501256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572714.501262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff92fc6704b100 x1875449863972864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.501270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449863972864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.501275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.501278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572714.501280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972864:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572714.501283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972992:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572714.501293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572714.501299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff92fc6704ad80 x1875449863972992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.501306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449863972992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.501311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.501313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572714.501316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863972992:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572714.501325:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973120:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572714.501335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572714.501341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572714/real 1788572714] req@ffff92fc6704a680 x1875449863973120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572714.501348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449863973120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572714.501354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572714.501356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572714.501358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973120:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788572714.948885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572715.012874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572715.972873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572716.036850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572716.996917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572717.060860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572718.020893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572718.084847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788572719.044847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572719.108820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572719.620825:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572719.620836:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572719.620840:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572719.620851:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4470 00000100:00080000:3.0:1788572719.620862:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572719.620869:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572719.620906:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863973248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572719.620940:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572719.620944:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572719.620946:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572719.620950:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4470 00000100:00080000:3.0:1788572719.620955:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572719.620960:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572719.620970:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572719.620978:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863973248:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572719.620986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863973376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572719.621000:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572719.621004:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572719.621005:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572719.621009:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4470 00000100:00080000:3.0:1788572719.621014:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572719.621018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572719.621024:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572719.621038:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863973504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572719.621048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572719.621052:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572719.621058:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572719.621060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572719.621064:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4470 00000100:00080000:3.0:1788572719.621068:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572719.621071:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572719.621075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572719.621089:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff92fc67338a80 x1875449863973248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.621090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863973632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572719.621125:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572719.621129:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572719.621130:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572719.621134:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863973248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.621136:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4470 00000100:00080000:3.0:1788572719.621142:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572719.621148:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788572719.621151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572719.621152:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572719.621157:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572719.621162:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863973760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572719.621167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863973248:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572719.621201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973376:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572719.621225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572719.621235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff92fc6733b100 x1875449863973376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572719.621245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863973376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572719.621251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572719.621254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572719.621256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973376:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572719.621260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973504:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572719.621274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572719.621285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff92fc67338380 x1875449863973504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572719.621311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863973504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572719.621320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572719.621324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572719.621327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973504:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572719.621332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973632:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572719.621348:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572719.621357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff92fc6733bb80 x1875449863973632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572719.621364:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863973632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572719.621370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572719.621372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572719.621375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973632:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572719.621378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973760:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572719.621388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572719.621394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff92fc67338000 x1875449863973760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572719.621401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863973760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572719.621406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572719.621408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572719.621410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863973760:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572720.068887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572720.132814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572721.092984:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572721.156875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572722.116902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572722.180840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572723.140875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572723.204830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572724.164934:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572724.228829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572724.740867:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.740877:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572724.740883:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.740896:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4475 00000100:00080000:2.0:1788572724.740911:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.740921:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572724.740968:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449863973888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572724.740998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.741002:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572724.741004:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.741008:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4475 00000100:00080000:2.0:1788572724.741012:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.741016:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572724.741027:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572724.741041:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449863974016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572724.741053:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863973888:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572724.741067:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.741076:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572724.741078:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.741084:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4475 00000100:00080000:2.0:1788572724.741086:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.741089:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572724.741093:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572724.741103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449863974144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572724.741111:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.741114:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572724.741115:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.741118:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4475 00000100:00080000:2.0:1788572724.741129:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.741132:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572724.741138:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572724.741144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572724.741153:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449863974272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572724.741164:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572724.741168:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572724.741170:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572724.741173:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4475 00000100:00080000:2.0:1788572724.741176:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572724.741179:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572724.741183:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572724.741196:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449863974400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572724.741197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff92fc4617ed80 x1875449863973888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.741257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449863973888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.741267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.741271:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572724.741275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863973888:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572724.741297:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974016:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572724.741318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572724.741332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff92fc4617c700 x1875449863974016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.741345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449863974016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.741355:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.741359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572724.741363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974016:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572724.741369:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974144:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572724.741386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572724.741395:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff92fc4617f100 x1875449863974144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.741402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449863974144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.741408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.741410:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572724.741412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974144:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572724.741416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974272:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572724.741427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572724.741433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff92fc4617f800 x1875449863974272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.741440:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449863974272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.741444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.741446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572724.741449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974272:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572724.741453:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974400:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572724.741464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572724.741469:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff92fc4617f480 x1875449863974400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.741477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449863974400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.741482:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.741483:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572724.741485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974400:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572725.188899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572725.252823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572726.212902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572726.276792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572727.236908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572727.300967:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572728.260837:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572728.324798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572729.284842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572729.348793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572729.860821:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572729.860830:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572729.860833:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572729.860841:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4480 00000100:00080000:0.0:1788572729.860850:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572729.860857:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572729.860889:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449863974528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572729.860912:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572729.860915:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572729.860917:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572729.860920:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4480 00000100:00080000:0.0:1788572729.860922:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572729.860925:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572729.860934:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572729.860943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449863974656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572729.860951:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572729.860956:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788572729.860956:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863974528:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572729.860958:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572729.860961:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4480 00000100:00080000:0.0:1788572729.860966:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572729.860970:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572729.860974:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572729.860981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863974784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572729.860988:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572729.860990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572729.860992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572729.860994:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4480 00000100:00080000:0.0:1788572729.860997:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572729.860999:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572729.861002:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572729.861010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449863974912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572729.861017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572729.861019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572729.861021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572729.861023:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4480 00000100:00080000:0.0:1788572729.861025:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572729.861028:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572729.861030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572729.861044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449863975040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572729.861110:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974656:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572729.861141:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974784:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572729.861157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974912:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572729.861172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975040:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572730.308900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572730.372866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572731.332883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572731.396874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572732.356849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572732.420874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788572732.933285:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:3.0:1788572732.933339:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572732.933359:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572732.933426:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572732.933438:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572732.933447:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572732.933453:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788572732.933771:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572732] req@ffff92fc6704aa00 x1875449863974656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572732.933799:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449863974656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572732.933814:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572732.933820:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572732.933825:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974656:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572732.933834:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572732] req@ffff92fc6704a300 x1875449863974784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572732.933847:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449863974784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572732.933857:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572732.933862:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572732.933865:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974784:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572732.933872:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572732] req@ffff92fc67049880 x1875449863974912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572732.933886:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449863974912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572732.933896:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572732.933900:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572732.933903:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863974912:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572732.933910:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572732] req@ffff92fc67048000 x1875449863975040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572732.933922:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449863975040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572732.933930:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572732.933934:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572732.933937:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975040:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572732.933943:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572732] req@ffff92fc67049f80 x1875449863974528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572732.933956:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449863974528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572732.933965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572732.933969:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572732.933972:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863974528:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572733.380864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572733.444822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572734.404847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572734.468847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572734.980880:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572734.980895:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572734.980901:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572734.980913:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4485 00000100:00080000:0.0:1788572734.980927:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572734.980937:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572734.980976:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449863975168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572734.981020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572734.981025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572734.981031:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572734.981036:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4485 00000100:00080000:0.0:1788572734.981040:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572734.981045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572734.981058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572734.981077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449863975296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572734.981090:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863975168:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572734.981093:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572734.981101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572734.981104:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572734.981109:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4485 00000100:00080000:0.0:1788572734.981113:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572734.981117:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572734.981124:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572734.981140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449863975424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572734.981151:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572734.981155:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572734.981158:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572734.981162:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4485 00000100:00080000:0.0:1788572734.981169:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572734.981174:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572734.981181:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572734.981183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572734.981198:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449863975552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572734.981209:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572734.981214:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572734.981215:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572734.981219:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4485 00000100:00080000:0.0:1788572734.981223:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572734.981227:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572734.981233:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572734.981246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff92fc67048a80 x1875449863975168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572734.981248:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449863975680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572734.981298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449863975168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572734.981313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572734.981319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572734.981324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863975168:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572734.981373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975296:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572734.981403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572734.981422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff92fc67048380 x1875449863975296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572734.981439:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449863975296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572734.981449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572734.981453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572734.981457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975296:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572734.981465:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975424:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572734.981487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572734.981499:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff92fc67048700 x1875449863975424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572734.981512:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449863975424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572734.981528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572734.981532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572734.981536:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975424:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572734.981542:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975552:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572734.981566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572734.981594:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff92fc67049f80 x1875449863975552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572734.981606:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449863975552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572734.981615:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572734.981619:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572734.981623:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975552:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572734.981630:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975680:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572734.981653:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572734.981665:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff92fc67048000 x1875449863975680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572734.981679:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449863975680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572734.981687:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572734.981691:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572734.981694:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975680:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572735.428834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572735.492837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572736.452882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572736.516890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572737.476851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572737.540858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572738.500884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572738.564906:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572739.524910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572739.588918:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572740.100928:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572740.100935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572740.100939:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572740.100956:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4490 00000100:00080000:2.0:1788572740.100966:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572740.100974:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572740.101004:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863975808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572740.101026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572740.101030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572740.101031:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572740.101036:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4490 00000100:00080000:2.0:1788572740.101040:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572740.101047:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572740.101060:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572740.101078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449863975936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572740.101083:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863975808:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572740.101120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572740.101123:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572740.101126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572740.101140:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4490 00000100:00080000:2.0:1788572740.101144:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572740.101149:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572740.101156:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572740.101169:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572740.101172:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863976064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572740.101183:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572740.101186:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572740.101187:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572740.101193:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4490 00000100:00080000:2.0:1788572740.101214:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572740.101220:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572740.101226:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572740.101239:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449863976192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572740.101250:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572740.101254:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572740.101255:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572740.101259:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4490 00000100:00080000:2.0:1788572740.101264:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572740.101269:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572740.101275:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572740.101286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572740/real 1788572740] req@ffff92fc4617ca80 x1875449863975808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572740.101292:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449863976320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572740.101493:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449863975808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572740.101509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572740.101515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572740.101520:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863975808:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572740.101549:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975936:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572740.101570:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572740.101581:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572740/real 1788572740] req@ffff92fc4617dc00 x1875449863975936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572740.101590:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449863975936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572740.101596:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572740.101598:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572740.101601:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863975936:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572740.101605:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976064:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572740.101616:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572740.101622:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572740/real 1788572740] req@ffff92fc4617e680 x1875449863976064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572740.101630:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449863976064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572740.101635:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572740.101637:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572740.101639:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976064:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572740.101642:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976192:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572740.101652:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572740.101658:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572740/real 1788572740] req@ffff92fc4617d180 x1875449863976192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572740.101666:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449863976192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572740.101672:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572740.101674:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572740.101676:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976192:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572740.101679:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976320:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572740.101689:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572740.101694:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572740/real 1788572740] req@ffff92fc4617c380 x1875449863976320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572740.101701:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449863976320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572740.101706:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572740.101708:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572740.101710:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976320:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572740.548873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572740.612911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572741.572874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572741.636883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572742.596894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572742.660842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572743.620909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572743.684830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572744.644851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572744.708861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572745.220877:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572745.220889:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572745.220894:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572745.220905:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4495 00000100:00080000:2.0:1788572745.220916:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572745.220924:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572745.220973:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449863976448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572745.221019:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572745.221024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572745.221026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572745.221034:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4495 00000100:00080000:2.0:1788572745.221038:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572745.221041:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572745.221052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572745.221064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449863976576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572745.221088:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572745.221094:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572745.221096:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572745.221101:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863976448:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572745.221102:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4495 00000100:00080000:2.0:1788572745.221107:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572745.221112:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572745.221117:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572745.221128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449863976704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572745.221135:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572745.221138:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572745.221139:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572745.221142:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4495 00000100:00080000:2.0:1788572745.221144:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572745.221147:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572745.221150:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572745.221158:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449863976832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572745.221164:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572745.221167:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572745.221169:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572745.221171:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4495 00000100:00080000:2.0:1788572745.221173:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572745.221178:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788572745.221179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788572745.221185:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572745.221194:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449863976960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572745.221232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff92fc4617ea00 x1875449863976448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572745.221250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449863976448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572745.221259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572745.221263:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572745.221266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863976448:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572745.221289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976576:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572745.221305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572745.221315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff92fc4617c000 x1875449863976576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572745.221324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449863976576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572745.221330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572745.221332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572745.221335:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976576:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572745.221339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976704:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572745.221350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572745.221356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff92fc4617d880 x1875449863976704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572745.221363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449863976704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572745.221369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572745.221371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572745.221373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976704:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572745.221378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976832:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572745.221390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572745.221396:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff92fc4617e300 x1875449863976832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572745.221404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449863976832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572745.221410:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572745.221413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572745.221416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976832:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572745.221422:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976960:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572745.221443:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572745.221454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572745/real 1788572745] req@ffff92fc4617fb80 x1875449863976960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572745.221467:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449863976960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572745.221476:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572745.221479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572745.221483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863976960:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572745.668868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572745.732830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572746.692897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572746.756807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572747.716850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572747.780857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572748.740933:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572748.804842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572749.764882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572749.828880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572750.340884:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.340895:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572750.340900:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.340910:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4500 00000100:00080000:2.0:1788572750.340922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.340929:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572750.340978:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449863977088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.341015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.341052:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572750.341055:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.341061:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4500 00000100:00080000:2.0:1788572750.341066:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.341071:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572750.341083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572750.341101:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863977088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572750.341108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449863977216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.341123:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.341126:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572750.341128:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.341132:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4500 00000100:00080000:2.0:1788572750.341136:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.341139:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572750.341144:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572750.341164:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449863977344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.341171:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.341174:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572750.341176:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.341179:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4500 00000100:00080000:2.0:1788572750.341182:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.341185:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572750.341189:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572750.341191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572750.341201:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449863977472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572750.341209:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572750.341212:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572750.341214:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572750.341217:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4500 00000100:00080000:2.0:1788572750.341219:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572750.341222:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572750.341225:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572750.341233:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449863977600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572750.341262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff92fc4617d500 x1875449863977088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.341281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449863977088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.341292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.341296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572750.341300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863977088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572750.341325:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977216:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572750.341345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572750.341355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff92fc4617ce00 x1875449863977216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.341366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449863977216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.341376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.341380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572750.341383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977216:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572750.341391:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977344:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572750.341412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572750.341424:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff92fc4617fb80 x1875449863977344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.341433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449863977344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.341439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.341441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572750.341443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977344:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572750.341447:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977472:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572750.341461:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572750.341468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff92fc4617e300 x1875449863977472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.341475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449863977472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.341480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.341482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572750.341485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977472:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572750.341488:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977600:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572750.341498:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572750.341504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572750/real 1788572750] req@ffff92fc4617d880 x1875449863977600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572750.341511:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449863977600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572750.341520:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572750.341522:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572750.341525:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977600:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572750.788902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572750.852840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572751.812835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572751.876794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572752.836872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572752.900881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572753.860867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572753.924863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572754.884856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572754.948840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572755.460831:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572755.460842:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572755.460847:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572755.460858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4506 00000100:00080000:0.0:1788572755.460869:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572755.460878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572755.460917:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449863977728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572755.460947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572755.460957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572755.460959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572755.460964:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4506 00000100:00080000:0.0:1788572755.460969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572755.460975:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572755.460986:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572755.460988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863977728:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572755.461006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863977856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572755.461019:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572755.461022:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572755.461024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572755.461040:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4506 00000100:00080000:0.0:1788572755.461045:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572755.461047:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572755.461052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572755.461063:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449863977984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572755.461067:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788572755.461086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572755.461090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572755.461091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572755.461097:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4506 00000100:00080000:0.0:1788572755.461102:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572755.461106:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572755.461113:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572755.461115:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff92fc67049880 x1875449863977728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572755.461131:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863978112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572755.461135:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449863977728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.461153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572755.461157:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572755.461162:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572755.461162:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788572755.461167:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572755.461172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863977728:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572755.461173:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4506 00000100:00080000:0.0:1788572755.461179:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572755.461184:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572755.461191:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572755.461201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977856:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572755.461203:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863978240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572755.461238:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572755.461253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff92fc6704a300 x1875449863977856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.461265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449863977856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.461274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.461277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572755.461281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977856:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572755.461287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977984:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572755.461305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572755.461316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff92fc6704aa00 x1875449863977984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.461326:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449863977984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.461334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.461337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572755.461340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863977984:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572755.461345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978112:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572755.461364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572755.461376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff92fc6704a680 x1875449863978112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.461387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449863978112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.461394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.461397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572755.461401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978112:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572755.461418:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978240:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572755.461439:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572755.461451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572755/real 1788572755] req@ffff92fc6704ad80 x1875449863978240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572755.461464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449863978240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572755.461473:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572755.461476:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572755.461478:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978240:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572755.908823:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572755.972847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572756.932854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572756.996834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572757.956824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572758.020859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572758.980929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572759.044882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572760.004951:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572760.068818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572760.580900:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.580909:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572760.580913:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.580922:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4511 00000100:00080000:2.0:1788572760.580931:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.580937:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572760.580970:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449863978368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572760.580996:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.580999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572760.581001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.581003:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4511 00000100:00080000:2.0:1788572760.581006:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.581008:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572760.581015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572760.581023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449863978496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572760.581030:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.581032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572760.581045:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.581049:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4511 00000100:00080000:2.0:1788572760.581051:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.581054:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572760.581058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572760.581067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449863978624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572760.581076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.581079:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572760.581080:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.581083:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4511 00000100:00080000:2.0:1788572760.581085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.581087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572760.581091:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572760.581102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449863978752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572760.581115:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863978368:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572760.581120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572760.581126:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572760.581127:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572760.581131:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4511 00000100:00080000:2.0:1788572760.581134:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572760.581138:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572760.581142:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572760.581156:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863978880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572760.581203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572760.581265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff92fc4617c000 x1875449863978368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.581284:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449863978368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.581296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.581300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572760.581304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863978368:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572760.581310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978496:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572760.581326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572760.581336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff92fc4617ea00 x1875449863978496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.581344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449863978496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.581350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.581352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572760.581354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978496:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572760.581358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978624:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572760.581369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572760.581375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff92fc4617c380 x1875449863978624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.581382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449863978624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.581387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.581389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572760.581391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978624:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572760.581427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978752:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572760.581444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572760.581453:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff92fc4617d180 x1875449863978752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.581461:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449863978752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.581466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.581468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572760.581471:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978752:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572760.581474:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978880:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572760.581484:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572760.581491:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff92fc4617e680 x1875449863978880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.581498:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449863978880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.581503:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.581505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572760.581507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863978880:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572761.028942:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572761.092872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572762.052859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572762.116858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572763.076921:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572763.140898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572764.100863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572764.164804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572765.124858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572765.188830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572765.700857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.700869:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572765.700874:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.700885:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4516 00000100:00080000:2.0:1788572765.700896:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.700903:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572765.700941:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449863979008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572765.700978:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.700982:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572765.700984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.700988:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4516 00000100:00080000:2.0:1788572765.700992:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.700996:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572765.701006:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572765.701022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863979136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572765.701033:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572765.701038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863979008:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572765.701039:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572765.701057:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.701063:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4516 00000100:00080000:2.0:1788572765.701067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.701070:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572765.701076:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572765.701090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449863979264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572765.701107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.701110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572765.701112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.701117:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4516 00000800:00000100:3.0:1788572765.701121:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788572765.701125:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.701132:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572765.701138:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572765.701151:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449863979392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572765.701161:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.701164:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572765.701166:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.701171:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4516 00000100:00100000:3.0:1788572765.701176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff92fc4617dc00 x1875449863979008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572765.701177:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.701186:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572765.701195:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572765.701209:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449863979520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572765.701212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449863979008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572765.701238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572765.701244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572765.701249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863979008:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572765.701275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979136:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572765.701301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572765.701317:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff92fc4617ca80 x1875449863979136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572765.701332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449863979136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572765.701341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572765.701345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572765.701349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979136:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572765.701355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979264:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572765.701377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572765.701389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff92fc4617f480 x1875449863979264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572765.701403:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449863979264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572765.701418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572765.701422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572765.701426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979264:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572765.701432:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979392:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572765.701456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572765.701468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff92fc4617f800 x1875449863979392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572765.701482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449863979392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572765.701490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572765.701494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572765.701497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979392:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572765.701504:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979520:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572765.701527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572765.701539:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff92fc4617f100 x1875449863979520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572765.701552:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449863979520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572765.701560:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572765.701563:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572765.701567:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979520:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572766.148867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572766.212794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572767.172939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572767.236896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572768.196881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572768.260841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572769.220883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572769.284827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572770.244935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572770.308862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572770.820845:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572770.820856:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572770.820861:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572770.820872:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4521 00000100:00080000:2.0:1788572770.820885:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572770.820892:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572770.820929:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449863979648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572770.820961:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572770.820966:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572770.820968:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572770.820972:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4521 00000100:00080000:2.0:1788572770.820976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572770.820981:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572770.820991:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572770.821004:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449863979776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572770.821012:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572770.821015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572770.821017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572770.821020:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4521 00000100:00080000:2.0:1788572770.821024:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572770.821027:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572770.821031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572770.821040:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449863979904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572770.821060:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572770.821070:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788572770.821072:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863979648:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572770.821073:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572770.821089:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4521 00000100:00080000:2.0:1788572770.821092:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572770.821095:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572770.821101:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572770.821117:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449863980032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572770.821124:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572770.821126:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572770.821128:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572770.821131:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4521 00000100:00080000:2.0:1788572770.821147:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572770.821150:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572770.821153:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572770.821168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572770.821169:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449863980160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572770.821231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff92fc4617c700 x1875449863979648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572770.821252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449863979648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572770.821262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572770.821265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572770.821268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863979648:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572770.821275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979776:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572770.821295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572770.821305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff92fc4617ed80 x1875449863979776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572770.821313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449863979776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572770.821319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572770.821322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572770.821325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979776:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572770.821349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979904:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572770.821361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572770.821368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff92fc4617f100 x1875449863979904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572770.821376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449863979904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572770.821381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572770.821383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572770.821385:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863979904:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572770.821389:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980032:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572770.821401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572770.821408:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff92fc4617f800 x1875449863980032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572770.821415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449863980032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572770.821420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572770.821422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572770.821424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980032:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572770.821427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980160:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572770.821438:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572770.821445:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff92fc4617f480 x1875449863980160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572770.821453:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449863980160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572770.821458:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572770.821459:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572770.821461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980160:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572771.268867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572771.332836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572772.292855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572772.356806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572773.316819:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572773.380822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572774.340859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572774.404828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572775.364818:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572775.428816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572775.940870:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.940880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572775.940884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.940895:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4526 00000100:00080000:2.0:1788572775.940905:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.940913:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572775.940955:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863980288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572775.940984:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.940987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572775.940989:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.940992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4526 00000100:00080000:2.0:1788572775.940994:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.940997:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572775.941006:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572775.941021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449863980416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572775.941031:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863980288:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572775.941039:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.941044:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572775.941045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.941049:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4526 00000100:00080000:2.0:1788572775.941052:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.941072:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572775.941076:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572775.941089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863980544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572775.941096:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.941099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572775.941101:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.941106:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4526 00000100:00080000:2.0:1788572775.941118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.941122:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572775.941127:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572775.941143:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449863980672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572775.941150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788572775.941166:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.941170:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572775.941172:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.941177:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4526 00000100:00080000:2.0:1788572775.941182:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.941185:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572775.941189:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572775.941201:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449863980800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572775.941222:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff92fc4617ca80 x1875449863980288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.941246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449863980288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.941256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.941261:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572775.941264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863980288:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572775.941291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980416:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572775.941310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572775.941321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff92fc4617dc00 x1875449863980416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.941331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449863980416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.941337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.941339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572775.941342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980416:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572775.941346:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980544:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572775.941357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572775.941364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff92fc4617e680 x1875449863980544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.941372:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449863980544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.941378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.941380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572775.941382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980544:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572775.941387:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980672:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572775.941401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572775.941414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff92fc4617d180 x1875449863980672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.941422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449863980672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.941428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.941431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572775.941433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980672:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572775.941437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980800:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572775.941452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572775.941461:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff92fc4617c380 x1875449863980800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.941470:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449863980800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.941476:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.941479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572775.941482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863980800:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572776.388850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572776.452852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572777.412863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572777.476788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572778.436860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572778.500837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572779.460902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572779.524823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572780.484847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572780.548836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572781.060886:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572781.060896:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572781.060900:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572781.060911:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4531 00000100:00080000:0.0:1788572781.060922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572781.060928:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572781.060961:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449863980928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572781.060994:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572781.060997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572781.060998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572781.061001:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4531 00000100:00080000:0.0:1788572781.061003:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572781.061005:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572781.061013:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572781.061022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449863981056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572781.061030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572781.061032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572781.061034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572781.061036:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4531 00000100:00080000:0.0:1788572781.061038:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572781.061040:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572781.061043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572781.061051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449863981184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572781.061059:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572781.061077:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788572781.061078:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863980928:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572781.061079:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572781.061092:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4531 00000100:00080000:0.0:1788572781.061096:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572781.061100:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572781.061104:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572781.061117:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449863981312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572781.061123:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572781.061126:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572781.061128:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572781.061131:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4531 00000100:00080000:0.0:1788572781.061133:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572781.061136:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572781.061140:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572781.061148:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449863981440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572781.061167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572781.061224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572781/real 1788572781] req@ffff92fc6704b100 x1875449863980928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572781.061257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449863980928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572781.061272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572781.061277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572781.061282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863980928:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572781.061289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981056:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572781.061310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572781.061321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572781/real 1788572781] req@ffff92fc6704b480 x1875449863981056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572781.061329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449863981056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572781.061335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572781.061338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572781.061340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981056:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572781.061364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981184:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572781.061376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572781.061383:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572781/real 1788572781] req@ffff92fc6704b800 x1875449863981184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572781.061391:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449863981184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572781.061396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572781.061399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572781.061401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981184:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572781.061404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981312:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572781.061416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572781.061423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572781/real 1788572781] req@ffff92fc67049c00 x1875449863981312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572781.061431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449863981312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572781.061436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572781.061439:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572781.061441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981312:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572781.061444:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981440:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572781.061454:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572781.061460:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572781/real 1788572781] req@ffff92fc67049180 x1875449863981440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572781.061467:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449863981440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572781.061472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572781.061474:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572781.061476:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981440:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572781.508858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572781.572824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572782.532868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572782.596867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572783.556909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572783.620817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572784.580834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572784.644832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572785.604878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572785.668789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572786.180827:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572786.180838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572786.180842:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572786.180853:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4536 00000100:00080000:1.0:1788572786.180864:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572786.180872:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572786.180904:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449863981568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572786.180932:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572786.180935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572786.180937:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572786.180940:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4536 00000100:00080000:1.0:1788572786.180943:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572786.180947:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572786.180956:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572786.180966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449863981696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572786.180971:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863981568:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572786.180979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572786.180982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572786.180983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572786.180986:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4536 00000100:00080000:1.0:1788572786.180988:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572786.180990:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572786.180994:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572786.181003:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449863981824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572786.181009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572786.181011:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572786.181013:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572786.181016:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4536 00000100:00080000:1.0:1788572786.181019:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572786.181021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572786.181025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572786.181032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449863981952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572786.181039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572786.181043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788572786.181044:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572786.181048:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572786.181051:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4536 00000100:00080000:1.0:1788572786.181055:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572786.181057:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572786.181061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572786.181083:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449863982080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572786.181106:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff92fc67288380 x1875449863981568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572786.181122:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449863981568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572786.181130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572786.181132:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572786.181135:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863981568:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572786.181154:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981696:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572786.181170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572786.181179:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff92fc67289c00 x1875449863981696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572786.181188:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449863981696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572786.181194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572786.181196:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572786.181198:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981696:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572786.181202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981824:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572786.181212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572786.181219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff92fc67289500 x1875449863981824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572786.181226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449863981824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572786.181236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572786.181238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572786.181241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981824:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572786.181244:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981952:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572786.181254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572786.181260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff92fc6728bb80 x1875449863981952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572786.181267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449863981952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572786.181272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572786.181273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572786.181275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863981952:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572786.181279:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982080:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572786.181288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572786.181294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572786/real 1788572786] req@ffff92fc6728b100 x1875449863982080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572786.181301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449863982080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572786.181306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572786.181308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572786.181310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982080:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572786.628862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572786.692849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572787.652889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572787.716865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572788.676874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572788.740833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572789.700863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572789.764820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572790.724813:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572790.788841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572791.300838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572791.300846:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572791.300850:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572791.300859:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4541 00000100:00080000:1.0:1788572791.300868:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572791.300875:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572791.300909:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449863982208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572791.300939:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572791.300942:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572791.300944:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572791.300946:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4541 00000100:00080000:1.0:1788572791.300949:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572791.300951:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572791.300959:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572791.300967:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449863982336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572791.300975:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572791.300977:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572791.300979:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572791.300981:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4541 00000100:00080000:1.0:1788572791.300983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572791.300985:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572791.300988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572791.300996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449863982464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572791.300999:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863982208:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572791.301005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572791.301009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572791.301011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572791.301013:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4541 00000100:00080000:1.0:1788572791.301015:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572791.301017:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572791.301020:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572791.301029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449863982592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572791.301035:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572791.301037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572791.301039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572791.301041:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4541 00000100:00080000:1.0:1788572791.301044:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572791.301047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572791.301050:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572791.301057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449863982720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572791.301088:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572791.301135:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff92fc67288700 x1875449863982208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572791.301162:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449863982208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572791.301174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572791.301178:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572791.301183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863982208:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572791.301191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982336:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572791.301219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572791.301233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff92fc67288a80 x1875449863982336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572791.301246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449863982336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572791.301255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572791.301259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572791.301263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982336:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572791.301294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982464:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572791.301317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572791.301331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff92fc67289180 x1875449863982464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572791.301344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449863982464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572791.301352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572791.301356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572791.301359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982464:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572791.301366:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982592:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572791.301388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572791.301400:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff92fc6728b100 x1875449863982592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572791.301413:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449863982592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572791.301422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572791.301426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572791.301429:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982592:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572791.301436:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982720:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572791.301456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572791.301469:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572791/real 1788572791] req@ffff92fc6728bb80 x1875449863982720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572791.301481:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449863982720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572791.301489:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572791.301492:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572791.301496:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982720:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572791.748856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572791.812878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572792.772816:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572792.836828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572793.796852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572793.860832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572794.820848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572794.884796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572795.844890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572795.908853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572796.420812:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.420824:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572796.420828:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.420839:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4547 00000100:00080000:2.0:1788572796.420850:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.420858:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572796.420893:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449863982848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.420917:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.420924:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572796.420926:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.420930:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4547 00000100:00080000:2.0:1788572796.420934:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.420938:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572796.420949:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572796.420961:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863982848:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572796.420962:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449863982976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.420974:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.420977:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572796.420979:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.420983:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4547 00000100:00080000:2.0:1788572796.420986:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.420990:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572796.420995:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572796.421014:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449863983104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.421022:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.421025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572796.421027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.421031:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4547 00000100:00080000:2.0:1788572796.421034:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.421038:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572796.421044:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572796.421053:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449863983232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572796.421063:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572796.421066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572796.421068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572796.421071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4547 00000100:00100000:3.0:1788572796.421071:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982976:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788572796.421075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572796.421080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572796.421101:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572796.421114:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449863983360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572796.421120:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983104:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572796.421147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983232:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572796.421175:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983360:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572796.868831:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572796.932841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572797.892821:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572797.956842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572798.916843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572798.980853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572799.493284:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788572799.493325:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572799.493342:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572799.493508:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572799.493521:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572799.493533:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572799.493543:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788572799.493564:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff92fc4617fb80 x1875449863983360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.493595:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449863983360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.493608:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.493614:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572799.493618:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983360:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572799.493626:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff92fc4617c000 x1875449863982976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.493637:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449863982976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.493645:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.493647:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572799.493650:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863982976:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572799.493656:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff92fc4617d880 x1875449863983104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.493667:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449863983104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.493673:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.493677:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572799.493679:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983104:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572799.493686:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff92fc4617e300 x1875449863983232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.493695:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449863983232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.493702:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.493705:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572799.493708:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983232:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572799.493714:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572796/real 1788572799] req@ffff92fc4617ea00 x1875449863982848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572799.493752:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449863982848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572799.493761:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572799.493765:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572799.493770:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863982848:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572799.940844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572800.004836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572800.964880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572801.028863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572801.540836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.540844:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572801.540848:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.540857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4552 00000100:00080000:2.0:1788572801.540865:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.540870:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572801.540901:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449863983488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.540922:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.540926:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572801.540928:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.540931:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4552 00000100:00080000:2.0:1788572801.540935:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.540938:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572801.540947:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572801.540961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449863983616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.540971:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.540973:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572801.540975:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.540978:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4552 00000100:00080000:2.0:1788572801.540982:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.540985:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572801.540990:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572801.541000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863983488:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572801.541009:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449863983744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.541022:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.541025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572801.541027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.541032:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4552 00000100:00080000:2.0:1788572801.541035:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.541040:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572801.541046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572801.541058:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449863983872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572801.541070:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572801.541072:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572801.541074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572801.541076:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4552 00000100:00080000:2.0:1788572801.541079:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572801.541085:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788572801.541086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788572801.541107:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572801.541123:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449863984000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572801.541163:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff92fc4617ce00 x1875449863983488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.541188:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449863983488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.541202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.541207:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572801.541212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863983488:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572801.541221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983616:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572801.541248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572801.541261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff92fc4617d500 x1875449863983616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.541274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449863983616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.541281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.541286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572801.541289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572801.541323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983744:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572801.541344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572801.541355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff92fc4617ea00 x1875449863983744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.541365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449863983744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.541372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.541375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572801.541378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572801.541383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983872:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572801.541400:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572801.541411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff92fc4617e300 x1875449863983872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.541422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449863983872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.541430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.541433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572801.541437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863983872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572801.541443:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984000:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572801.541462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572801.541473:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572801/real 1788572801] req@ffff92fc4617d880 x1875449863984000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572801.541483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449863984000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572801.541490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572801.541493:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572801.541495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984000:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572801.988883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572802.052887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572803.012883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572803.076856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572804.036857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572804.100853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572805.060876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572805.124782:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572806.084840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572806.148895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572806.660893:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.660902:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572806.660906:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.660919:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4557 00000100:00080000:0.0:1788572806.660930:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.660938:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572806.660977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449863984128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572806.661001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.661005:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572806.661007:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.661011:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4557 00000100:00080000:0.0:1788572806.661014:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.661019:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572806.661029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572806.661043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449863984256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572806.661048:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863984128:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572806.661053:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.661056:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572806.661057:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.661061:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4557 00000100:00080000:0.0:1788572806.661064:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.661068:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572806.661072:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572806.661080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449863984384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572806.661093:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.661111:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572806.661112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.661116:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4557 00000100:00080000:0.0:1788572806.661120:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.661123:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572806.661128:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572806.661131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572806.661138:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449863984512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572806.661146:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.661149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572806.661151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.661154:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4557 00000100:00080000:0.0:1788572806.661156:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.661160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572806.661164:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572806.661175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff92fc6704bb80 x1875449863984128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572806.661186:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449863984640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572806.661194:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449863984128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572806.661204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572806.661208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572806.661212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863984128:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572806.661236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984256:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572806.661258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572806.661268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff92fc67048e00 x1875449863984256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572806.661281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449863984256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572806.661287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572806.661289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572806.661291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984256:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572806.661295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984384:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572806.661306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572806.661312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff92fc67049500 x1875449863984384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572806.661320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449863984384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572806.661325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572806.661327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572806.661328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984384:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572806.661332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984512:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572806.661343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572806.661348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff92fc67049180 x1875449863984512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572806.661356:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449863984512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572806.661365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572806.661367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572806.661369:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984512:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572806.661372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984640:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572806.661381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572806.661386:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff92fc67049c00 x1875449863984640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572806.661393:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449863984640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572806.661398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572806.661400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572806.661402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984640:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572807.108890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572807.172810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572808.132849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572808.196907:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572809.156973:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572809.220900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572810.180868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572810.244851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572811.204870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572811.268799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572811.780857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572811.780868:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572811.780873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572811.780886:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4562 00000100:00080000:2.0:1788572811.780897:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572811.780907:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572811.780957:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449863984768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572811.780986:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572811.780990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572811.780992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572811.781000:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4562 00000100:00080000:2.0:1788572811.781008:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572811.781015:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572811.781030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572811.781035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863984768:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572811.781062:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449863984896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572811.781076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572811.781079:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572811.781081:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572811.781086:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4562 00000100:00080000:2.0:1788572811.781091:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572811.781096:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572811.781114:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788572811.781128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572811.781130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449863985024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572811.781143:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572811.781147:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572811.781150:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572811.781159:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4562 00000100:00080000:2.0:1788572811.781164:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572811.781168:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572811.781175:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572811.781193:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449863985152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572811.781197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff92fc4617c000 x1875449863984768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572811.781212:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572811.781226:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572811.781229:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572811.781238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449863984768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572811.781240:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4562 00000100:00080000:2.0:1788572811.781248:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572811.781256:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788572811.781264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788572811.781264:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572811.781288:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449863985280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572811.781290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572811.781300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863984768:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572811.781335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984896:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572811.781364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572811.781380:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff92fc4617fb80 x1875449863984896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572811.781396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449863984896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572811.781406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572811.781410:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572811.781415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863984896:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572811.781422:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985024:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572811.781445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572811.781460:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff92fc4617c380 x1875449863985024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572811.781482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449863985024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572811.781492:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572811.781496:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572811.781501:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985024:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572811.781509:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985152:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572811.781533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572811.781548:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff92fc4617d180 x1875449863985152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572811.781563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449863985152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572811.781573:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572811.781578:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572811.781581:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985152:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572811.781590:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985280:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572811.781621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572811.781638:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572811] req@ffff92fc4617e680 x1875449863985280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572811.781653:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449863985280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572811.781669:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572811.781672:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572811.781676:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985280:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572812.228845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572812.292848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572813.252898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572813.316823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572814.276851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572814.340856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572815.300971:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572815.364829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572816.324878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572816.388794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572816.900826:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572816.900834:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572816.900837:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572816.900846:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4567 00000100:00080000:0.0:1788572816.900854:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572816.900860:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572816.900891:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449863985408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572816.900913:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572816.900916:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572816.900917:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572816.900920:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4567 00000100:00080000:0.0:1788572816.900923:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572816.900933:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572816.900946:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572816.900967:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863985408:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572816.900971:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449863985536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572816.900985:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572816.900987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572816.900989:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572816.900992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4567 00000100:00080000:0.0:1788572816.900995:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572816.900997:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572816.901001:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572816.901010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449863985664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572816.901016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572816.901019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572816.901020:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572816.901022:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4567 00000100:00080000:0.0:1788572816.901024:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572816.901026:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788572816.901030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788572816.901031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572816.901041:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863985792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572816.901048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572816.901050:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572816.901051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572816.901054:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4567 00000100:00080000:0.0:1788572816.901057:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572816.901059:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572816.901063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572816.901070:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff92fc6704b800 x1875449863985408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572816.901073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863985920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572816.901114:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449863985408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.901123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.901127:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572816.901130:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863985408:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572816.901149:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985536:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572816.901170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572816.901184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff92fc6704b480 x1875449863985536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.901222:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449863985536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.901229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.901231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572816.901233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985536:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572816.901238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985664:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572816.901254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572816.901263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff92fc6704b100 x1875449863985664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.901270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449863985664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.901276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.901278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572816.901280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985664:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572816.901284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985792:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572816.901295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572816.901301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff92fc6704ad80 x1875449863985792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.901309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449863985792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.901314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.901316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572816.901318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985792:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572816.901321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985920:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572816.901331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572816.901337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff92fc6704a680 x1875449863985920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.901344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449863985920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.901349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.901351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572816.901353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863985920:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572817.348858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572817.412832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572818.372871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572818.436848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572819.396871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572819.460839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572820.420855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572820.484831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572821.444872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572821.508834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572822.020875:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572822.020886:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572822.020891:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572822.020901:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4572 00000100:00080000:1.0:1788572822.020910:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572822.020916:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572822.020959:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449863986048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572822.020979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572822.020983:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572822.020985:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572822.020989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4572 00000100:00080000:1.0:1788572822.020993:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572822.020998:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572822.021008:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863986048:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788572822.021009:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572822.021028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449863986176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572822.021041:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572822.021045:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572822.021047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572822.021052:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4572 00000100:00080000:1.0:1788572822.021066:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572822.021071:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788572822.021075:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572822.021081:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572822.021092:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449863986304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572822.021100:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572822.021102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572822.021104:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572822.021107:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4572 00000100:00080000:1.0:1788572822.021110:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572822.021113:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572822.021118:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572822.021119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572822/real 1788572822] req@ffff92fc67289500 x1875449863986048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572822.021128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449863986432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572822.021139:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572822.021142:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572822.021143:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572822.021148:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4572 00000100:00100000:3.0:1788572822.021148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449863986048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572822.021150:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572822.021157:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572822.021163:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572822.021173:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449863986560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572822.021175:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572822.021183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572822.021190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863986048:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572822.021212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986176:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572822.021232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572822.021242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572822/real 1788572822] req@ffff92fc67289c00 x1875449863986176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572822.021254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449863986176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572822.021263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572822.021267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572822.021270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986176:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572822.021277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986304:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572822.021299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572822.021335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572822/real 1788572822] req@ffff92fc67288380 x1875449863986304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572822.021347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449863986304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572822.021355:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572822.021359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572822.021362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986304:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572822.021370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986432:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572822.021401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572822.021411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572822/real 1788572822] req@ffff92fc6728a680 x1875449863986432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572822.021419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449863986432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572822.021425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572822.021427:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572822.021429:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986432:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572822.021433:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986560:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572822.021452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572822.021462:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572822/real 1788572822] req@ffff92fc6728b800 x1875449863986560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572822.021475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449863986560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572822.021483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572822.021486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572822.021489:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986560:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572822.468840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572822.532831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572823.492928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572823.556904:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572824.516923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572824.580917:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572825.540828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572825.604817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572826.564938:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572826.628832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572827.140862:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572827.140871:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572827.140874:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572827.140884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4577 00000100:00080000:0.0:1788572827.140893:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572827.140899:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572827.140953:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449863986688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572827.140981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572827.140985:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572827.140987:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572827.140993:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4577 00000100:00080000:0.0:1788572827.140997:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572827.141000:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572827.141009:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572827.141021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863986816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572827.141030:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863986688:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572827.141031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572827.141038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572827.141041:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572827.141045:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4577 00000100:00080000:0.0:1788572827.141047:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572827.141049:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572827.141053:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572827.141061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449863986944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572827.141067:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572827.141070:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572827.141071:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572827.141074:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4577 00000100:00080000:0.0:1788572827.141077:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572827.141079:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572827.141082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572827.141089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449863987072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572827.141096:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572827.141098:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572827.141099:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572827.141101:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4577 00000100:00080000:0.0:1788572827.141104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572827.141108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788572827.141112:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572827.141116:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572827.141124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449863987200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572827.141154:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572827/real 1788572827] req@ffff92fc6704aa00 x1875449863986688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572827.141171:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449863986688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572827.141179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572827.141182:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572827.141185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863986688:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572827.141204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986816:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572827.141220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572827.141228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572827/real 1788572827] req@ffff92fc6704a300 x1875449863986816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572827.141237:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449863986816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572827.141243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572827.141245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572827.141255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986816:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572827.141259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986944:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572827.141271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572827.141279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572827/real 1788572827] req@ffff92fc67049880 x1875449863986944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572827.141287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449863986944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572827.141297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572827.141300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572827.141302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863986944:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572827.141305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987072:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572827.141316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572827.141322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572827/real 1788572827] req@ffff92fc67048000 x1875449863987072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572827.141329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449863987072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572827.141334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572827.141336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572827.141338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987072:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572827.141341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987200:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572827.141356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572827.141367:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572827/real 1788572827] req@ffff92fc67049f80 x1875449863987200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572827.141380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449863987200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572827.141389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572827.141392:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572827.141396:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987200:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572827.588887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572827.652839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572828.612853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572828.676850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572829.636830:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572829.700852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572830.660887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572830.724892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572831.684847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572831.748838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572832.260892:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.260907:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572832.260913:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.260926:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4582 00000100:00080000:2.0:1788572832.260940:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.260949:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572832.261009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449863987328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572832.261055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.261061:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572832.261065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.261071:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4582 00000100:00080000:2.0:1788572832.261077:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.261084:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572832.261097:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572832.261114:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449863987456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572832.261133:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.261141:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572832.261144:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572832.261144:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863987328:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572832.261150:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4582 00000100:00080000:2.0:1788572832.261159:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.261165:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572832.261169:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572832.261181:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449863987584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572832.261190:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.261194:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572832.261196:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.261199:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4582 00000100:00080000:2.0:1788572832.261218:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.261222:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572832.261227:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572832.261240:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449863987712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572832.261243:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788572832.261259:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572832.261263:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572832.261265:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572832.261269:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4582 00000100:00080000:2.0:1788572832.261273:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572832.261277:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572832.261282:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572832.261291:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449863987840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572832.261319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff92fc4617dc00 x1875449863987328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.261346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449863987328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.261359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.261364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572832.261368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863987328:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572832.261397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987456:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572832.261427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572832.261444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff92fc4617ca80 x1875449863987456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.261458:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449863987456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.261467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.261471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572832.261475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987456:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572832.261482:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987584:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572832.261540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572832.261555:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff92fc4617f480 x1875449863987584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.261568:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449863987584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.261577:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.261580:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572832.261584:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987584:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572832.261591:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987712:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572832.261625:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572832.261638:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff92fc4617f800 x1875449863987712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.261652:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449863987712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.261661:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.261664:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572832.261668:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987712:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572832.261675:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987840:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572832.261696:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572832.261707:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572832/real 1788572832] req@ffff92fc4617f100 x1875449863987840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572832.261719:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449863987840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572832.261758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572832.261762:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572832.261766:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863987840:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572832.708880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572832.772863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572833.732900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572833.796852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572834.756872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572834.820905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572835.780909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572835.844814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572836.804868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572836.868848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572837.380884:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.380896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572837.380901:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.380913:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4587 00000100:00080000:0.0:1788572837.380927:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.380935:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572837.381010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449863987968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572837.381048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.381053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572837.381055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.381059:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4587 00000100:00080000:0.0:1788572837.381062:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.381066:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572837.381079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572837.381091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449863988096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572837.381101:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.381105:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572837.381106:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.381109:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4587 00000100:00080000:0.0:1788572837.381112:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.381115:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572837.381120:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572837.381128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449863988224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572837.381138:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.381141:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572837.381142:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.381148:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4587 00000100:00080000:0.0:1788572837.381151:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.381155:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572837.381176:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572837.381190:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449863988352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572837.381190:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863987968:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572837.381207:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572837.381210:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572837.381211:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572837.381214:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4587 00000100:00080000:0.0:1788572837.381228:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572837.381230:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572837.381234:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572837.381242:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449863988480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572837.381300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572837.381350:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff92fc67048700 x1875449863987968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.381367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449863987968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.381378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.381382:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572837.381386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863987968:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572837.381392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988096:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572837.381410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572837.381450:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff92fc67048380 x1875449863988096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.381462:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449863988096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.381470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.381473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572837.381477:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988096:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572837.381483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988224:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788572837.381500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572837.381509:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff92fc67048a80 x1875449863988224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.381520:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449863988224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.381528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.381530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572837.381533:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988224:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572837.381562:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988352:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572837.381578:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572837.381588:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff92fc67049f80 x1875449863988352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.381598:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449863988352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.381606:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.381632:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572837.381635:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988352:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788572837.381641:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988480:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788572837.381687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572837.381700:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572837/real 1788572837] req@ffff92fc67048000 x1875449863988480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572837.381717:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449863988480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572837.381727:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572837.381732:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572837.381777:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988480:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572837.828858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572837.892920:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572838.852825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572838.916800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572839.876809:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572839.940812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572840.900879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572840.964879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572841.924929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572841.988799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572842.500881:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572842.500890:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572842.500893:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572842.500902:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4593 00000100:00080000:3.0:1788572842.500911:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572842.500918:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572842.500952:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863988608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572842.500980:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572842.500983:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572842.500984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572842.500987:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4593 00000100:00080000:3.0:1788572842.500990:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572842.500993:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572842.501001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572842.501018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863988736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572842.501033:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863988608:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572842.501037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572842.501054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572842.501057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572842.501062:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4593 00000100:00080000:3.0:1788572842.501064:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572842.501067:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572842.501070:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572842.501078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863988864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572842.501084:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572842.501087:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572842.501088:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572842.501090:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4593 00000100:00080000:3.0:1788572842.501092:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572842.501095:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572842.501098:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572842.501103:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863988992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572842.501114:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572842.501117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572842.501118:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572842.501121:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4593 00000100:00080000:3.0:1788572842.501125:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572842.501128:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572842.501131:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572842.501137:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572842.501142:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863989120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572842.501197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff92fc67339500 x1875449863988608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572842.501221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863988608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572842.501231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572842.501235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572842.501238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863988608:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572842.501266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988736:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572842.501293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572842.501308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff92fc6733b480 x1875449863988736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572842.501323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863988736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572842.501329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572842.501332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572842.501334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988736:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572842.501339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988864:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572842.501356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572842.501363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff92fc6733a300 x1875449863988864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572842.501371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863988864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572842.501381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572842.501384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572842.501386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988864:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572842.501389:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988992:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572842.501402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572842.501408:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff92fc6733b800 x1875449863988992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572842.501414:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863988992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572842.501419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572842.501421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572842.501424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863988992:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572842.501428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989120:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572842.501439:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572842.501444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572842/real 1788572842] req@ffff92fc67338700 x1875449863989120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572842.501451:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863989120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572842.501456:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572842.501458:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572842.501461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989120:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572842.948905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572843.012822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572843.972964:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572844.036834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572844.996984:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572845.060892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572846.020986:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572846.084919:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572847.044929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572847.108872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572847.620942:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.620956:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572847.620963:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.620975:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4598 00000100:00080000:3.0:1788572847.620989:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.620998:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572847.621062:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449863989248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.621105:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.621109:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572847.621110:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.621114:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4598 00000100:00080000:3.0:1788572847.621118:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.621138:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572847.621150:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572847.621164:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449863989376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.621173:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.621181:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572847.621183:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.621204:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4598 00000100:00080000:3.0:1788572847.621211:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572847.621218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863989248:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572847.621218:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572847.621226:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572847.621244:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449863989504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.621274:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.621278:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572847.621280:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.621284:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4598 00000100:00080000:3.0:1788572847.621286:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.621289:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572847.621295:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572847.621316:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863989632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.621323:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.621326:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572847.621327:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.621330:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4598 00000800:00000100:0.0:1788572847.621349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572847.621394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff92fc6733a680 x1875449863989248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572847.621413:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449863989248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572847.621423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572847.621435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572847.621439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863989248:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572847.621448:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989376:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572847.621471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572847.621491:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff92fc67338e00 x1875449863989376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572847.621500:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449863989376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572847.621505:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572847.621508:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572847.621511:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989376:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572847.621531:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989504:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572847.621548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572847.621556:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff92fc6733aa00 x1875449863989504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572847.621565:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449863989504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572847.621572:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572847.621576:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572847.621581:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788572847.621587:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572847.621591:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989504:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572847.621604:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989632:192.168.202.132@tcp:8:kworker.0 00000008:00100000:3.0:1788572847.621604:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572847.621632:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863989760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572847.621639:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572847.621650:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff92fc67338700 x1875449863989632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572847.621663:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863989632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572847.621676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572847.621681:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572847.621688:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989632:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572847.621780:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989760:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572847.621829:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572847.621847:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff92fc6733b800 x1875449863989760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572847.621865:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863989760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572847.621871:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572847.621874:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572847.621876:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863989760:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572848.068932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572848.132942:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572849.092820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572849.156870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572850.116842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572850.180848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572851.140826:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572851.204863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572852.164864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572852.228873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572852.740788:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572852.740797:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572852.740801:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572852.740809:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4603 00000100:00080000:3.0:1788572852.740818:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572852.740824:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572852.740862:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863989888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572852.740889:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572852.740894:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572852.740896:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572852.740901:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4603 00000100:00080000:3.0:1788572852.740905:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572852.740910:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572852.740933:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572852.740935:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863989888:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572852.740948:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863990016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572852.740957:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572852.740960:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572852.740962:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572852.740965:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4603 00000100:00080000:3.0:1788572852.740968:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572852.740972:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572852.740976:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572852.740983:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863990144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572852.740990:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572852.740993:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572852.740996:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788572852.740997:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572852.741008:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4603 00000100:00080000:3.0:1788572852.741013:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572852.741016:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572852.741020:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572852.741029:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863990272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572852.741036:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff92fc6733a300 x1875449863989888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572852.741038:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572852.741048:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572852.741052:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572852.741057:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4603 00000100:00080000:3.0:1788572852.741059:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572852.741063:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863989888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572852.741064:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572852.741072:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788572852.741083:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788572852.741085:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863990400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.741090:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572852.741098:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863989888:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572852.741126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990016:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572852.741143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572852.741152:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff92fc6733b480 x1875449863990016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572852.741161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863990016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572852.741167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572852.741169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572852.741172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990016:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572852.741176:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990144:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572852.741195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572852.741202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff92fc67339500 x1875449863990144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572852.741210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863990144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572852.741215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572852.741217:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572852.741219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990144:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572852.741223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990272:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572852.741234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572852.741240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff92fc67338000 x1875449863990272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572852.741247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863990272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572852.741253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572852.741255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572852.741257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990272:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572852.741260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990400:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572852.741271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572852.741276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff92fc6733bb80 x1875449863990400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572852.741283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863990400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572852.741289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572852.741290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572852.741293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990400:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572853.188822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572853.252830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572854.212835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572854.276857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572855.236895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572855.300828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572856.260842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572856.324828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572857.284889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572857.348851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572857.860875:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572857.860886:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572857.860891:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572857.860902:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4608 00000100:00080000:3.0:1788572857.860914:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572857.860922:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572857.860974:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863990528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572857.861008:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572857.861011:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572857.861030:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572857.861034:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4608 00000100:00080000:3.0:1788572857.861038:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572857.861041:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572857.861052:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572857.861069:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863990656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572857.861079:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572857.861084:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788572857.861085:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863990528:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572857.861086:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572857.861092:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4608 00000100:00080000:3.0:1788572857.861098:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572857.861100:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572857.861105:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572857.861115:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863990784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572857.861125:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572857.861129:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572857.861131:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572857.861136:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4608 00000100:00080000:3.0:1788572857.861141:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572857.861146:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572857.861153:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572857.861167:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449863990912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572857.861170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572857.861180:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572857.861184:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572857.861188:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572857.861193:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4608 00000100:00080000:3.0:1788572857.861198:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572857.861203:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572857.861209:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572857.861223:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449863991040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572857.861253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff92fc67338380 x1875449863990528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572857.861278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863990528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.861292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572857.861297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572857.861301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863990528:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572857.861330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990656:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572857.861354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572857.861369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff92fc6733b100 x1875449863990656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572857.861383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863990656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.861393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572857.861397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572857.861400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990656:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572857.861407:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990784:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572857.861426:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572857.861438:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff92fc67338a80 x1875449863990784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572857.861451:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863990784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.861461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572857.861464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572857.861468:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990784:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572857.861475:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990912:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572857.861494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572857.861506:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff92fc67339180 x1875449863990912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572857.861519:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449863990912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.861528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572857.861532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572857.861536:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863990912:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572857.861542:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991040:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572857.861561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572857.861571:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff92fc67339f80 x1875449863991040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572857.861584:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449863991040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.861593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572857.861596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572857.861600:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991040:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572858.308913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572858.372909:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572859.332857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572859.396821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572860.356822:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572860.420857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572861.380892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572861.444809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572862.404867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572862.468804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572862.980852:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572862.980860:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572862.980865:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572862.980876:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4613 00000100:00080000:3.0:1788572862.980887:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572862.980895:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572862.980934:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863991168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572862.980958:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572862.980967:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572862.980969:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572862.980972:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4613 00000100:00080000:3.0:1788572862.980975:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572862.980979:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572862.980988:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572862.981002:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449863991296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572862.981003:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863991168:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572862.981016:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572862.981019:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572862.981021:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572862.981024:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4613 00000100:00080000:3.0:1788572862.981026:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572862.981029:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572862.981032:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572862.981040:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863991424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572862.981052:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572862.981056:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572862.981058:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572862.981062:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4613 00000100:00080000:3.0:1788572862.981066:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572862.981077:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572862.981084:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572862.981099:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449863991552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572862.981108:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991296:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788572862.981114:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572862.981132:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572862.981134:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572862.981138:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4613 00000100:00080000:3.0:1788572862.981142:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572862.981146:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572862.981151:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572862.981155:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991424:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788572862.981165:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449863991680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572862.981191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991552:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572862.981221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991680:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572863.428855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572863.492818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572864.452866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572864.516895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572865.476864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572865.540847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788572866.052949:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788572866.052986:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572866.052999:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788572866.053033:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788572866.053041:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788572866.053046:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788572866.053053:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788572866.053165:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572866] req@ffff92fc67339f80 x1875449863991552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572866.053207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449863991552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572866.053221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572866.053227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572866.053231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991552:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572866.053240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572866] req@ffff92fc67339180 x1875449863991680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572866.053261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449863991680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572866.053269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572866.053273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572866.053277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991680:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572866.053284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572866] req@ffff92fc6733ad80 x1875449863991296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572866.053298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449863991296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572866.053307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572866.053311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572866.053315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991296:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572866.053328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572866] req@ffff92fc67339c00 x1875449863991424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572866.053343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449863991424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572866.053351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572866.053354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572866.053358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991424:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572866.053365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572866] req@ffff92fc67339880 x1875449863991168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572866.053376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449863991168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572866.053384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572866.053388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572866.053391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863991168:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572866.500814:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572866.564820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572867.524889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572867.588870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572868.100857:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572868.100874:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572868.100880:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572868.100893:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4618 00000100:00080000:3.0:1788572868.100909:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572868.100920:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572868.100968:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863991808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572868.101004:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572868.101008:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572868.101010:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572868.101016:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4618 00000100:00080000:3.0:1788572868.101020:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572868.101024:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572868.101038:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572868.101055:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863991936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572868.101068:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572868.101073:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572868.101075:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572868.101081:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4618 00000100:00080000:3.0:1788572868.101086:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572868.101091:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572868.101098:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572868.101117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863991808:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572868.101117:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863992064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572868.101151:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572868.101156:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572868.101158:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572868.101164:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4618 00000100:00080000:3.0:1788572868.101170:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572868.101176:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572868.101184:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572868.101204:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863992192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572868.101237:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572868.101242:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572868.101244:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572868.101249:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4618 00000100:00080000:3.0:1788572868.101257:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572868.101263:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572868.101269:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572868.101287:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863992320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572868.101302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572868.101394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572868/real 1788572868] req@ffff92fc67338a80 x1875449863991808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572868.101421:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863991808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572868.101436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572868.101442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572868.101446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863991808:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572868.101458:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991936:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572868.101487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572868.101504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572868/real 1788572868] req@ffff92fc6733b100 x1875449863991936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572868.101518:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863991936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572868.101526:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572868.101530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572868.101533:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863991936:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572868.101571:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992064:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572868.101596:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572868.101609:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572868/real 1788572868] req@ffff92fc67338380 x1875449863992064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572868.101622:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863992064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572868.101630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572868.101656:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572868.101660:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992064:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572868.101666:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992192:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572868.101690:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572868.101701:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572868/real 1788572868] req@ffff92fc6733bb80 x1875449863992192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572868.101713:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863992192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572868.101772:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572868.101776:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572868.101780:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992192:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572868.101788:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992320:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572868.101810:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572868.101820:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572868/real 1788572868] req@ffff92fc67338000 x1875449863992320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572868.101834:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863992320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572868.101842:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572868.101845:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572868.101848:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992320:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572868.548896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572868.612873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572869.572936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572869.636888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572870.596868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572870.660916:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572871.620849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572871.684881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572872.644858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572872.708860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572873.220872:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572873.220885:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572873.220889:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572873.220900:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4623 00000100:00080000:3.0:1788572873.220912:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572873.220920:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572873.220966:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863992448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572873.220994:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572873.220997:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572873.220999:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572873.221002:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4623 00000100:00080000:3.0:1788572873.221004:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572873.221007:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572873.221018:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572873.221031:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863992576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572873.221039:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572873.221042:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572873.221043:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572873.221046:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4623 00000100:00080000:3.0:1788572873.221048:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572873.221051:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572873.221056:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572873.221059:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863992448:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572873.221072:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863992704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572873.221086:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572873.221090:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572873.221093:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572873.221098:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4623 00000100:00080000:3.0:1788572873.221103:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572873.221109:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572873.221115:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572873.221133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572873.221134:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863992832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572873.221149:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572873.221153:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572873.221155:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572873.221160:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4623 00000100:00080000:3.0:1788572873.221165:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572873.221171:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572873.221177:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572873.221182:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff92fc67339500 x1875449863992448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572873.221194:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863992960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572873.221203:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863992448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572873.221220:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572873.221224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572873.221228:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863992448:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572873.221234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992576:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572873.221252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572873.221277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff92fc6733b480 x1875449863992576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572873.221293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863992576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572873.221303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572873.221308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572873.221312:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992576:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572873.221364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992704:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572873.221382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572873.221391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff92fc6733a300 x1875449863992704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572873.221399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863992704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572873.221406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572873.221408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572873.221410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992704:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572873.221415:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992832:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572873.221427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572873.221433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff92fc6733b800 x1875449863992832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572873.221448:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863992832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572873.221459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572873.221461:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572873.221463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992832:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572873.221467:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992960:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572873.221479:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572873.221486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572873/real 1788572873] req@ffff92fc67338700 x1875449863992960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572873.221493:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863992960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572873.221498:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572873.221500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572873.221502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863992960:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572873.668953:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572873.732897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572874.692930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572874.756875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572875.716880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572875.780856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572876.740853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572876.804817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572877.764863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572877.828786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572878.340849:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572878.340857:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572878.340861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572878.340870:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4628 00000100:00080000:1.0:1788572878.340879:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572878.340884:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572878.340916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449863993088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572878.340944:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572878.340949:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572878.340951:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572878.340955:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4628 00000100:00080000:1.0:1788572878.340967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572878.340975:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572878.340986:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572878.340998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863993088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572878.341005:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449863993216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572878.341020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572878.341024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572878.341027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572878.341031:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4628 00000100:00080000:1.0:1788572878.341051:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572878.341058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788572878.341058:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572878.341072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572878.341086:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449863993344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572878.341094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572878.341097:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572878.341100:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572878.341101:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff92fc67288e00 x1875449863993088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572878.341106:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4628 00000100:00080000:1.0:1788572878.341113:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572878.341118:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572878.341122:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572878.341130:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449863993088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572878.341134:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449863993472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572878.341165:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572878.341168:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572878.341170:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572878.341170:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572878.341182:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:1.0:1788572878.341182:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4628 00000100:00080000:1.0:1788572878.341188:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572878.341191:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863993088:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572878.341195:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572878.341200:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572878.341211:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449863993600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572878.341225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993216:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572878.341256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572878.341271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff92fc67289f80 x1875449863993216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572878.341285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449863993216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572878.341294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572878.341298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572878.341300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993216:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572878.341305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993344:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572878.341324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572878.341336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff92fc67288000 x1875449863993344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572878.341349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449863993344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572878.341358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572878.341363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572878.341367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993344:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572878.341374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993472:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572878.341391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572878.341423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff92fc67289880 x1875449863993472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572878.341433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449863993472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572878.341439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572878.341441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572878.341443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993472:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572878.341448:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993600:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572878.341462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572878.341470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572878/real 1788572878] req@ffff92fc6728aa00 x1875449863993600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572878.341477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449863993600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572878.341482:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572878.341484:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572878.341486:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993600:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572878.788889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572878.852793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572879.812884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572879.876840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572880.836859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572880.900819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572881.860923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572881.924853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572882.884829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572882.948811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572883.460866:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572883.460875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572883.460879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572883.460889:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4634 00000100:00080000:3.0:1788572883.460899:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572883.460907:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572883.460948:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449863993728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572883.460985:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572883.460992:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572883.460994:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572883.460998:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4634 00000100:00080000:3.0:1788572883.461001:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572883.461004:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572883.461014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572883.461027:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449863993856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572883.461035:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572883.461037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572883.461039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572883.461043:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863993728:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572883.461044:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4634 00000100:00080000:3.0:1788572883.461049:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572883.461053:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572883.461056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572883.461065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449863993984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572883.461072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572883.461075:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572883.461076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572883.461079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4634 00000100:00080000:3.0:1788572883.461113:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572883.461119:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572883.461126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572883.461132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572883.461143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863994112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572883.461151:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572883.461154:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572883.461155:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572883.461158:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4634 00000100:00080000:3.0:1788572883.461162:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572883.461185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff92fc6733aa00 x1875449863993728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.461218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449863993728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572883.461222:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572883.461228:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572883.461238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788572883.461241:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863994240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572883.461247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572883.461255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863993728:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572883.461285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993856:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572883.461312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572883.461327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff92fc67338e00 x1875449863993856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.461340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449863993856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.461349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.461353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572883.461358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993856:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572883.461365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993984:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572883.461390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572883.461403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff92fc6733a680 x1875449863993984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.461417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449863993984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.461425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.461428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572883.461431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863993984:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572883.461437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994112:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572883.461474:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572883.461491:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff92fc67338700 x1875449863994112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.461503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863994112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.461510:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.461513:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572883.461517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994112:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572883.461523:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994240:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572883.461542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572883.461553:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572883/real 1788572883] req@ffff92fc6733b800 x1875449863994240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572883.461565:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863994240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572883.461580:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572883.461583:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572883.461586:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994240:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572883.908871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572883.972879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572884.932899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572884.996859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572885.956885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572886.020898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572886.980895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572887.044922:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572888.004892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572888.068864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572888.580875:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572888.580889:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572888.580894:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572888.580905:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4639 00000100:00080000:3.0:1788572888.580919:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572888.580929:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572888.580978:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863994368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572888.581008:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572888.581013:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572888.581015:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572888.581019:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4639 00000100:00080000:3.0:1788572888.581024:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572888.581029:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572888.581040:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572888.581057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863994496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572888.581058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863994368:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572888.581075:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572888.581078:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572888.581080:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572888.581084:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4639 00000100:00080000:3.0:1788572888.581089:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572888.581093:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572888.581099:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572888.581110:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863994624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572888.581120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572888.581123:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572888.581126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572888.581130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4639 00000100:00080000:3.0:1788572888.581134:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572888.581138:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788572888.581143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788572888.581147:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572888.581160:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863994752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572888.581175:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572888.581178:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572888.581180:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572888.581183:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4639 00000100:00080000:3.0:1788572888.581187:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572888.581191:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572888.581197:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572888.581201:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff92fc6733a300 x1875449863994368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572888.581212:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863994880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572888.581226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863994368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.581238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.581243:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572888.581248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863994368:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572888.581277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994496:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572888.581298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572888.581310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff92fc6733b480 x1875449863994496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.581320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863994496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.581333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.581336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572888.581340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994496:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572888.581345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994624:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572888.581361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572888.581371:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff92fc67339500 x1875449863994624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.581381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863994624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.581388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.581391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572888.581394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994624:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572888.581400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994752:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572888.581416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572888.581425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff92fc67338000 x1875449863994752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.581436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863994752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.581442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.581445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572888.581449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994752:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572888.581454:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994880:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572888.581471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572888.581480:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff92fc6733bb80 x1875449863994880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572888.581490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863994880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572888.581497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572888.581500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572888.581503:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863994880:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572889.028854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572889.092845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572890.052840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572890.116882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572891.076946:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572891.140865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572892.100896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572892.164843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572893.124843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572893.188843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572893.700826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572893.700835:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572893.700839:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572893.700849:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4644 00000100:00080000:3.0:1788572893.700859:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572893.700866:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572893.700894:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863995008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572893.700918:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572893.700921:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572893.700922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572893.700924:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4644 00000100:00080000:3.0:1788572893.700927:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572893.700929:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572893.700936:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572893.700946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863995136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572893.700952:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572893.700955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572893.700955:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572893.700957:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4644 00000100:00080000:3.0:1788572893.700959:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572893.700961:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572893.700964:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572893.700966:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863995008:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572893.700972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863995264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572893.700979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572893.700981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572893.700982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572893.700984:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4644 00000100:00080000:3.0:1788572893.700985:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572893.700987:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572893.700990:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572893.700996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863995392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572893.701002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572893.701004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572893.701005:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572893.701008:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4644 00000100:00080000:3.0:1788572893.701018:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572893.701020:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572893.701023:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572893.701029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863995520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572893.701036:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572893.701090:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff92fc67338380 x1875449863995008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572893.701108:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863995008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572893.701116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572893.701120:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572893.701123:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863995008:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572893.701130:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995136:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572893.701148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572893.701173:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff92fc6733b100 x1875449863995136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572893.701184:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863995136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572893.701191:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572893.701194:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572893.701196:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995136:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572893.701220:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995264:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572893.701236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572893.701244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff92fc67338a80 x1875449863995264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572893.701251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863995264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572893.701256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572893.701258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572893.701260:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995264:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572893.701263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995392:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572893.701275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572893.701283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff92fc67339880 x1875449863995392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572893.701293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449863995392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572893.701301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572893.701304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572893.701307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995392:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572893.701313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995520:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572893.701330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572893.701341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff92fc67339c00 x1875449863995520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572893.701351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449863995520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572893.701359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572893.701362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572893.701365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995520:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572894.148851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572894.212824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572895.172845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572895.236836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572896.196877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572896.260796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572897.220870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572897.284798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572898.244891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572898.308818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572898.820839:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572898.820848:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572898.820852:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572898.820861:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4649 00000100:00080000:0.0:1788572898.820870:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572898.820877:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572898.820909:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449863995648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572898.820931:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572898.820934:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572898.820935:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572898.820938:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4649 00000100:00080000:0.0:1788572898.820940:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572898.820943:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572898.820952:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572898.820961:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449863995776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572898.820970:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572898.820971:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863995648:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572898.820975:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572898.820979:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572898.820982:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4649 00000100:00080000:0.0:1788572898.820985:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572898.820987:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572898.820990:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572898.820999:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449863995904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572898.821005:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572898.821008:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572898.821009:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572898.821011:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4649 00000100:00080000:0.0:1788572898.821014:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572898.821017:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572898.821020:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572898.821027:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449863996032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572898.821033:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572898.821035:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572898.821037:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572898.821039:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4649 00000100:00080000:0.0:1788572898.821042:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788572898.821046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788572898.821049:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572898.821053:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572898.821061:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449863996160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572898.821095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff92fc67049880 x1875449863995648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.821112:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449863995648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.821123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.821127:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572898.821132:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863995648:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572898.821158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995776:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572898.821179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572898.821189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff92fc6704a300 x1875449863995776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.821197:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449863995776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.821203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.821205:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572898.821207:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995776:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572898.821211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995904:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572898.821222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572898.821228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff92fc6704aa00 x1875449863995904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.821235:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449863995904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.821240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.821242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572898.821243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863995904:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572898.821247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996032:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572898.821258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572898.821264:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff92fc6704a680 x1875449863996032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.821271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449863996032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.821276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.821278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572898.821279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996032:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572898.821283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996160:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572898.821292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572898.821298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff92fc6704ad80 x1875449863996160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572898.821305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449863996160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572898.821310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572898.821312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572898.821314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996160:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572899.268842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572899.332838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572900.292854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572900.356825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572901.316838:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572901.380872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572902.340867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572902.404852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572903.364871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572903.428857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572903.940910:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572903.940925:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572903.940931:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572903.940944:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4654 00000100:00080000:3.0:1788572903.940958:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572903.940968:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572903.941022:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449863996288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572903.941062:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572903.941067:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572903.941069:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572903.941073:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4654 00000100:00080000:3.0:1788572903.941078:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572903.941083:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572903.941095:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572903.941111:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449863996416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572903.941125:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863996288:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572903.941136:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572903.941142:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572903.941144:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572903.941149:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4654 00000100:00080000:3.0:1788572903.941153:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572903.941157:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572903.941163:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572903.941178:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449863996544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572903.941188:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572903.941192:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572903.941194:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572903.941198:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4654 00000100:00080000:3.0:1788572903.941203:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572903.941207:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572903.941213:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572903.941219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572903.941231:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449863996672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572903.941243:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572903.941247:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572903.941249:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572903.941254:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4654 00000100:00080000:3.0:1788572903.941259:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572903.941264:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572903.941271:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572903.941287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff92fc6733ad80 x1875449863996288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572903.941298:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449863996800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572903.941316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449863996288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572903.941334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572903.941339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572903.941343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863996288:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572903.941368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996416:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572903.941388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572903.941399:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff92fc67339180 x1875449863996416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572903.941408:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449863996416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572903.941414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572903.941416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572903.941418:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996416:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572903.941422:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996544:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572903.941433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572903.941440:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff92fc67339f80 x1875449863996544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572903.941447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449863996544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572903.941453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572903.941455:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572903.941457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996544:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572903.941461:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996672:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572903.941473:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572903.941479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff92fc67339c00 x1875449863996672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572903.941486:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449863996672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572903.941492:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572903.941493:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572903.941495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996672:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572903.941499:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996800:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572903.941510:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572903.941515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff92fc67339880 x1875449863996800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572903.941522:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449863996800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572903.941528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572903.941530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572903.941532:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863996800:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572904.388911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572904.452904:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572905.412866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572905.476828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572906.436856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572906.500827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572907.460815:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572907.524834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572908.484946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572908.548885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572909.060809:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572909.060820:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572909.060825:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572909.060835:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4659 00000100:00080000:3.0:1788572909.060846:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572909.060855:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572909.060908:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449863996928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572909.060942:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572909.060947:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572909.060950:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572909.060954:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4659 00000100:00080000:3.0:1788572909.060959:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572909.060964:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572909.060977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572909.060993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449863997056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572909.060998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863996928:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572909.061009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572909.061015:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572909.061017:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572909.061020:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4659 00000100:00080000:3.0:1788572909.061024:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572909.061027:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572909.061032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572909.061041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449863997184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572909.061049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572909.061052:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572909.061053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572909.061056:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4659 00000100:00080000:3.0:1788572909.061067:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572909.061070:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572909.061073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572909.061081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449863997312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572909.061089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572909.061091:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572909.061095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572909.061097:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572909.061100:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4659 00000100:00080000:3.0:1788572909.061103:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572909.061105:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572909.061110:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572909.061118:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863997440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572909.061150:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572909/real 1788572909] req@ffff92fc67338a80 x1875449863996928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572909.061175:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449863996928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572909.061189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572909.061194:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572909.061199:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863996928:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572909.061227:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997056:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572909.061251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572909.061266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572909/real 1788572909] req@ffff92fc6733b100 x1875449863997056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572909.061281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449863997056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572909.061290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572909.061293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572909.061296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997056:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572909.061301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997184:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572909.061317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572909.061326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572909/real 1788572909] req@ffff92fc67338380 x1875449863997184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572909.061334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449863997184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572909.061344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572909.061347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572909.061349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997184:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572909.061352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997312:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572909.061364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572909.061370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572909/real 1788572909] req@ffff92fc6733bb80 x1875449863997312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572909.061377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449863997312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572909.061382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572909.061384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572909.061386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997312:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572909.061390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997440:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572909.061400:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572909.061406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572909/real 1788572909] req@ffff92fc67338000 x1875449863997440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572909.061412:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863997440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572909.061418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572909.061419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572909.061422:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997440:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788572909.508861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572909.572862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572910.532872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572910.596841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572911.556920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572911.620801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572912.580888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572912.644851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572913.604928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572913.668815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572914.180853:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.180861:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572914.180864:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.180873:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4664 00000100:00080000:1.0:1788572914.180882:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.180889:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572914.180921:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449863997568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572914.180950:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.180955:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572914.180956:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.180961:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4664 00000100:00080000:1.0:1788572914.180965:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.180969:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572914.180980:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572914.180994:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449863997696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572914.181000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863997568:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572914.181007:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.181010:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572914.181012:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.181015:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4664 00000100:00080000:1.0:1788572914.181017:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.181020:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572914.181025:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572914.181034:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449863997824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572914.181041:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.181044:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572914.181045:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.181047:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4664 00000100:00080000:1.0:1788572914.181050:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.181053:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572914.181056:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572914.181064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572914.181067:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449863997952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572914.181075:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572914.181077:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572914.181079:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572914.181082:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4664 00000100:00080000:1.0:1788572914.181084:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572914.181086:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572914.181089:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572914.181095:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449863998080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572914.181119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff92fc6728b480 x1875449863997568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572914.181136:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449863997568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572914.181148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572914.181152:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572914.181157:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863997568:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788572914.181181:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997696:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572914.181208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572914.181222:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff92fc6728ad80 x1875449863997696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572914.181231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449863997696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572914.181238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572914.181240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572914.181243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997696:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572914.181247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997824:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788572914.181261:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572914.181270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff92fc6728a300 x1875449863997824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572914.181296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449863997824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572914.181306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572914.181308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572914.181311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997824:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788572914.181315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997952:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572914.181336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572914.181349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff92fc6728aa00 x1875449863997952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572914.181370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449863997952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572914.181378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572914.181381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572914.181384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863997952:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788572914.181390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998080:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788572914.181411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788572914.181423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572914/real 1788572914] req@ffff92fc67289880 x1875449863998080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572914.181434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449863998080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572914.181442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572914.181445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788572914.181447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998080:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572914.628917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572914.692834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572915.652933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572915.716786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572916.676875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572916.740868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572917.700904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572917.764820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572918.724952:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572918.788908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572919.300818:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.300826:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572919.300829:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.300839:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4669 00000100:00080000:3.0:1788572919.300848:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.300864:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572919.300903:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863998208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572919.300929:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.300932:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572919.300934:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.300937:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4669 00000100:00080000:3.0:1788572919.300940:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.300943:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572919.300953:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572919.300965:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863998336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572919.300982:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863998208:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572919.300989:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.300995:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572919.300997:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.301004:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4669 00000100:00080000:3.0:1788572919.301010:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.301015:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572919.301021:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572919.301037:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863998464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572919.301048:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.301050:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572919.301052:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788572919.301056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572919.301059:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4669 00000100:00080000:3.0:1788572919.301064:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.301067:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572919.301072:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572919.301081:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863998592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572919.301088:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572919.301090:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572919.301092:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572919.301094:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4669 00000100:00080000:3.0:1788572919.301099:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572919.301104:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572919.301105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff92fc67339500 x1875449863998208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788572919.301110:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572919.301128:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863998720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572919.301136:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863998208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572919.301147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572919.301151:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572919.301155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863998208:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572919.301179:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998336:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572919.301197:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572919.301209:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff92fc6733b480 x1875449863998336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572919.301224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863998336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572919.301234:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572919.301238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572919.301242:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998336:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572919.301259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998464:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572919.301276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572919.301285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff92fc6733a300 x1875449863998464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572919.301293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863998464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572919.301298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572919.301300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572919.301303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998464:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572919.301307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998592:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572919.301317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572919.301323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff92fc6733b800 x1875449863998592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572919.301331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863998592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572919.301341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572919.301343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572919.301345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998592:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572919.301348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998720:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572919.301364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572919.301369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572919/real 1788572919] req@ffff92fc67338700 x1875449863998720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572919.301376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863998720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572919.301382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572919.301384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572919.301386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998720:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788572919.749013:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572919.812794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572920.772900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572920.836844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572921.796918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572921.860826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572922.820847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572922.884838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572923.844897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572923.908805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572924.420803:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.420815:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572924.420820:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.420830:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4675 00000100:00080000:3.0:1788572924.420841:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.420848:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572924.420886:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449863998848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572924.420929:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.420934:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572924.420936:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.420941:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4675 00000100:00080000:3.0:1788572924.420945:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.420950:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572924.420960:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572924.420972:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863998848:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572924.420978:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449863998976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572924.420993:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.420997:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572924.420999:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.421003:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4675 00000100:00080000:3.0:1788572924.421006:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.421011:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572924.421016:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572924.421027:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449863999104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572924.421037:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.421041:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572924.421043:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788572924.421047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572924.421050:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4675 00000100:00080000:3.0:1788572924.421055:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.421060:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572924.421065:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572924.421079:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449863999232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572924.421089:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff92fc6733a680 x1875449863998848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572924.421090:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572924.421100:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572924.421105:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572924.421111:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4675 00000100:00100000:1.0:1788572924.421115:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449863998848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572924.421118:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572924.421127:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:1.0:1788572924.421129:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572924.421136:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:3.0:1788572924.421137:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572924.421140:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863998848:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572924.421154:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449863999360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572924.421166:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998976:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572924.421186:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572924.421196:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff92fc67338e00 x1875449863998976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572924.421204:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449863998976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572924.421210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572924.421212:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572924.421214:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863998976:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572924.421218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999104:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572924.421229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572924.421235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff92fc6733aa00 x1875449863999104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572924.421242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449863999104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572924.421247:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572924.421248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572924.421250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999104:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572924.421253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999232:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572924.421264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572924.421269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff92fc67338700 x1875449863999232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572924.421278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449863999232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572924.421283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572924.421284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572924.421286:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999232:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572924.421295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999360:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572924.421305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572924.421311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572924/real 1788572924] req@ffff92fc6733b800 x1875449863999360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572924.421318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449863999360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572924.421323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572924.421325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572924.421326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999360:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572924.868858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572924.932808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572925.892898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572925.956837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572926.916879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572926.980845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572927.940928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572928.004869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572928.964808:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572929.028834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572929.540856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572929.540867:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572929.540872:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572929.540883:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4680 00000100:00080000:3.0:1788572929.540894:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572929.540902:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572929.540948:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449863999488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572929.541025:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572929.541030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572929.541033:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572929.541037:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4680 00000100:00080000:3.0:1788572929.541042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572929.541046:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572929.541057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572929.541073:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449863999616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572929.541092:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572929.541095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572929.541101:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572929.541102:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863999488:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572929.541107:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4680 00000100:00080000:3.0:1788572929.541114:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572929.541120:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572929.541124:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572929.541135:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449863999744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572929.541144:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572929.541148:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572929.541150:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572929.541154:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4680 00000100:00080000:3.0:1788572929.541156:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572929.541159:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572929.541163:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572929.541172:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449863999872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572929.541181:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572929.541184:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572929.541186:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572929.541189:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4680 00000100:00080000:3.0:1788572929.541194:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572929.541198:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572929.541202:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572929.541211:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864000000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572929.541276:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572929.541330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572929.541364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572929.541398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000000:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788572929.988938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572930.052826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572931.012946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572931.076863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572932.036926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572932.100841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788572932.613015:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788572932.613069:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788572932.613086:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572932.613129:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572932.613138:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572932.613147:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788572932.613156:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788572932.613195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff92fc6733b480 x1875449863999616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572932.613222:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449863999616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572932.613235:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572932.613241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572932.613245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572932.613254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff92fc67339500 x1875449863999744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572932.613266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449863999744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572932.613275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572932.613278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572932.613282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572932.613289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff92fc67338000 x1875449863999872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572932.613307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449863999872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572932.613316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572932.613320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572932.613324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449863999872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572932.613332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff92fc6733bb80 x1875449864000000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572932.613344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864000000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572932.613351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572932.613354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572932.613358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000000:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572932.613364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572929/real 1788572932] req@ffff92fc6733a300 x1875449863999488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572932.613383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449863999488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572932.613391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572932.613395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572932.613398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449863999488:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572933.060904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572933.124832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788572934.084919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572934.148839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572934.660825:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572934.660834:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572934.660837:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572934.660846:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4685 00000100:00080000:3.0:1788572934.660856:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572934.660863:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572934.660899:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864000128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572934.660931:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572934.660935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572934.660937:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572934.660942:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4685 00000100:00080000:3.0:1788572934.660946:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572934.660949:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572934.660959:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572934.660975:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864000256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572934.660988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864000128:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572934.660990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572934.660999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572934.661001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572934.661006:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4685 00000100:00080000:3.0:1788572934.661010:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572934.661014:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572934.661046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572934.661064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864000384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572934.661074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572934.661081:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572934.661085:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572934.661087:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572934.661093:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4685 00000100:00080000:3.0:1788572934.661099:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572934.661104:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572934.661120:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572934.661126:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff92fc67338380 x1875449864000128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572934.661136:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864000512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572934.661150:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572934.661154:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572934.661156:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572934.661171:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4685 00000100:00100000:1.0:1788572934.661172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864000128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572934.661177:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572934.661186:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572934.661193:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788572934.661194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572934.661203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572934.661208:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864000640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572934.661212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864000128:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572934.661253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000256:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572934.661283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572934.661299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff92fc6733b100 x1875449864000256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572934.661312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864000256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572934.661322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572934.661326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572934.661329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000256:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572934.661335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000384:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572934.661355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572934.661367:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff92fc67338a80 x1875449864000384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572934.661377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864000384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572934.661386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572934.661389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572934.661392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000384:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572934.661399:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000512:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572934.661430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572934.661441:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff92fc67339880 x1875449864000512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572934.661452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864000512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572934.661460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572934.661463:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572934.661466:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000512:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572934.661473:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000640:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572934.661491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572934.661503:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff92fc67339c00 x1875449864000640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572934.661521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864000640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572934.661531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572934.661534:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572934.661537:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000640:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572935.108877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572935.172811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572936.132895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572936.196823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572937.156876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572937.220796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572938.180863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572938.244831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572939.204855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572939.268787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572939.780798:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572939.780806:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572939.780810:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572939.780819:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4690 00000100:00080000:3.0:1788572939.780828:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572939.780833:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572939.780874:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864000768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572939.780901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572939.780905:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572939.780906:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572939.780914:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4690 00000100:00080000:3.0:1788572939.780918:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572939.780921:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572939.780928:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572939.780939:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864000896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572939.780946:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572939.780951:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572939.780952:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572939.780955:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4690 00000100:00080000:3.0:1788572939.780958:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572939.780960:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572939.780963:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572939.780969:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864001024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572939.780976:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864000768:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572939.780979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572939.780985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572939.780987:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572939.780990:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4690 00000100:00080000:3.0:1788572939.780992:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572939.780994:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572939.780997:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572939.781007:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864001152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572939.781013:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572939.781015:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572939.781016:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572939.781020:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4690 00000100:00080000:3.0:1788572939.781023:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572939.781025:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572939.781028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572939.781037:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864001280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572939.781044:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572939.781099:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff92fc67339f80 x1875449864000768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572939.781120:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864000768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572939.781127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572939.781131:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572939.781133:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864000768:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572939.781138:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000896:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572939.781155:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572939.781164:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff92fc67339180 x1875449864000896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572939.781177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864000896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572939.781183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572939.781185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572939.781187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864000896:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572939.781206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001024:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572939.781218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572939.781226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff92fc6733ad80 x1875449864001024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572939.781233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864001024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572939.781242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572939.781245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572939.781247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001024:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572939.781250:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001152:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572939.781260:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572939.781266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff92fc67339c00 x1875449864001152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572939.781272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864001152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572939.781278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572939.781279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572939.781281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001152:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572939.781285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001280:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572939.781297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572939.781307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff92fc67339880 x1875449864001280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572939.781318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864001280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572939.781327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572939.781330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572939.781334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001280:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572940.228856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572940.292816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572941.252916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572941.316836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572942.276844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572942.340822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572943.300855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572943.364842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572944.324830:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572944.388809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572944.900857:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572944.900869:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572944.900873:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572944.900885:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4695 00000100:00080000:3.0:1788572944.900897:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572944.900906:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572944.900946:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864001408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572944.900978:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572944.900982:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572944.900984:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572944.900989:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4695 00000100:00080000:3.0:1788572944.900993:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572944.900997:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572944.901007:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572944.901021:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864001536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572944.901032:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572944.901036:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572944.901039:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572944.901043:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4695 00000100:00080000:3.0:1788572944.901048:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572944.901053:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572944.901058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864001408:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788572944.901060:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572944.901093:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864001664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572944.901107:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572944.901111:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572944.901113:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572944.901118:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4695 00000100:00080000:3.0:1788572944.901123:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572944.901127:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572944.901133:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572944.901145:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572944.901149:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864001792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572944.901163:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572944.901178:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572944.901180:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572944.901186:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4695 00000100:00100000:1.0:1788572944.901188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff92fc67338a80 x1875449864001408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572944.901193:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572944.901203:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572944.901209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864001408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788572944.901210:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788572944.901223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572944.901227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788572944.901232:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864001920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572944.901233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864001408:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572944.901245:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001536:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572944.901268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572944.901279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff92fc6733b100 x1875449864001536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572944.901289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864001536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572944.901295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572944.901298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572944.901301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001536:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572944.901320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001664:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572944.901331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572944.901339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff92fc67338380 x1875449864001664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572944.901347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864001664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572944.901352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572944.901354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572944.901356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001664:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572944.901359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001792:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572944.901370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572944.901376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff92fc6733a300 x1875449864001792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572944.901387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864001792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572944.901394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572944.901398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572944.901401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001792:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572944.901405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001920:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572944.901421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572944.901432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572944] req@ffff92fc6733bb80 x1875449864001920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572944.901445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864001920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572944.901455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572944.901458:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572944.901462:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864001920:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572945.348870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572945.412834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572946.372854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572946.436828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572947.396904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572947.460823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572948.420843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572948.484828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572949.444920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572949.508870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572950.020834:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572950.020844:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572950.020849:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572950.020860:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4700 00000100:00080000:0.0:1788572950.020872:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572950.020880:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572950.020925:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864002048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572950.020954:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572950.020957:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572950.020959:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572950.020962:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4700 00000100:00080000:0.0:1788572950.020964:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572950.020966:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572950.020976:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572950.020987:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449864002176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572950.020994:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572950.020998:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572950.021000:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572950.021003:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4700 00000100:00080000:0.0:1788572950.021006:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572950.021014:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864002048:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572950.021017:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572950.021024:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572950.021032:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449864002304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572950.021039:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572950.021043:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572950.021044:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572950.021047:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4700 00000100:00080000:0.0:1788572950.021049:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572950.021051:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572950.021054:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572950.021078:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864002432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572950.021085:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572950.021087:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572950.021089:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572950.021091:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4700 00000100:00080000:0.0:1788572950.021109:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572950.021112:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572950.021115:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572950.021124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572950.021125:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864002560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572950.021196:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572950/real 1788572950] req@ffff92fc6704b100 x1875449864002048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572950.021220:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864002048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572950.021233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572950.021238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572950.021241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864002048:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572950.021248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002176:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572950.021268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572950.021279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572950/real 1788572950] req@ffff92fc6704b480 x1875449864002176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572950.021287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449864002176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572950.021293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572950.021295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572950.021298:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002176:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572950.021321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002304:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788572950.021333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572950.021340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572950/real 1788572950] req@ffff92fc6704b800 x1875449864002304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572950.021348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449864002304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572950.021354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572950.021356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572950.021359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002304:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572950.021362:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002432:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572950.021372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572950.021378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572950/real 1788572950] req@ffff92fc67049c00 x1875449864002432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572950.021385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864002432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572950.021391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572950.021392:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572950.021394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002432:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572950.021398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002560:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788572950.021408:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788572950.021414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572950/real 1788572950] req@ffff92fc67049180 x1875449864002560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572950.021422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864002560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572950.021427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572950.021429:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572950.021431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002560:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572950.468907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572950.532809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572951.492899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572951.556894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572952.516913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572952.580827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572953.540844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572953.604849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572954.564855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572954.628796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572955.140857:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572955.140868:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572955.140873:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572955.140886:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4705 00000100:00080000:3.0:1788572955.140900:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572955.140910:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572955.140957:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864002688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572955.140997:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572955.141002:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572955.141004:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572955.141008:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4705 00000100:00080000:3.0:1788572955.141013:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572955.141018:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572955.141058:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572955.141073:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864002816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572955.141083:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864002688:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572955.141086:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572955.141093:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572955.141095:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572955.141099:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4705 00000100:00080000:3.0:1788572955.141102:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572955.141106:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572955.141112:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572955.141123:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864002944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572955.141131:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572955.141134:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572955.141136:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572955.141139:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4705 00000100:00080000:3.0:1788572955.141142:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572955.141146:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572955.141149:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572955.141150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572955.141161:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864003072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572955.141182:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572955.141190:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572955/real 1788572955] req@ffff92fc67338000 x1875449864002688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572955.141209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864002688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572955.141221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572955.141226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572955.141230:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864002688:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572955.141257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002816:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788572955.141268:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572955.141271:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572955.141276:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4705 00000100:00080000:3.0:1788572955.141280:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572955.141287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572955.141287:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572955.141296:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572955.141312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572955/real 1788572955] req@ffff92fc67339500 x1875449864002816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572955.141327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864002816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572955.141344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572955.141348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572955.141356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002816:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572955.141364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002944:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572955.141395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572955.141408:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864003200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572955.141410:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572955/real 1788572955] req@ffff92fc6733b480 x1875449864002944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572955.141445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864002944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572955.141454:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572955.141457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572955.141461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864002944:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572955.141466:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003072:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572955.141487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572955.141505:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572955/real 1788572955] req@ffff92fc6733b800 x1875449864003072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572955.141515:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864003072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572955.141527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572955.141529:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572955.141531:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003072:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572955.141545:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003200:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572955.141564:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572955.141576:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572955/real 1788572955] req@ffff92fc67338700 x1875449864003200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572955.141588:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864003200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572955.141594:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572955.141596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572955.141598:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003200:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572955.588886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572955.652829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572956.612869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572956.676795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572957.636844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572957.700812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572958.660852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572958.724833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572959.684844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572959.748826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788572960.260847:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572960.260856:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572960.260860:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572960.260868:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4710 00000100:00080000:1.0:1788572960.260877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572960.260883:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788572960.260914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864003328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572960.260943:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572960.260946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572960.260948:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572960.260950:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4710 00000100:00080000:1.0:1788572960.260953:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572960.260958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572960.260968:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572960.260982:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864003456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572960.260990:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864003328:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788572960.261014:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572960.261032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572960.261035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572960.261041:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4710 00000100:00080000:1.0:1788572960.261045:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572960.261050:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572960.261057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572960.261070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864003584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572960.261080:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788572960.261086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788572960.261089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572960.261093:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572960.261099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4710 00000100:00080000:1.0:1788572960.261107:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572960.261111:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788572960.261116:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572960.261128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864003712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572960.261132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff92fc67288000 x1875449864003328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572960.261147:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572960.261151:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572960.261152:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572960.261156:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4710 00000100:00100000:0.0:1788572960.261159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864003328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572960.261161:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572960.261169:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788572960.261173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788572960.261174:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788572960.261181:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572960.261185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864003328:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788572960.261188:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864003840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572960.261209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003456:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572960.261229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572960.261237:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff92fc67289f80 x1875449864003456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572960.261246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864003456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572960.261252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572960.261254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572960.261256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003456:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572960.261260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003584:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572960.261271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572960.261277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff92fc67288e00 x1875449864003584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572960.261289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864003584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572960.261294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572960.261296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572960.261298:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003584:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572960.261302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003712:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572960.261312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572960.261318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff92fc6728b800 x1875449864003712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572960.261325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864003712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572960.261330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572960.261332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572960.261334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003712:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572960.261337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003840:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572960.261347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572960.261352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572960/real 1788572960] req@ffff92fc6728a680 x1875449864003840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572960.261360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864003840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572960.261369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572960.261371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572960.261373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864003840:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572960.708853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572960.772845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572961.732854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572961.796832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572962.756932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572962.820894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572963.780819:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572963.844905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572964.804888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572964.868868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788572965.380846:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572965.380857:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572965.380862:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572965.380872:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4715 00000100:00080000:2.0:1788572965.380884:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572965.380892:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788572965.380941:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864003968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572965.380974:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572965.380978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572965.380980:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572965.380983:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4715 00000100:00080000:2.0:1788572965.380988:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572965.380992:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572965.381025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572965.381039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864004096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572965.381050:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572965.381052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864003968:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788572965.381054:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572965.381062:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572965.381077:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4715 00000100:00080000:2.0:1788572965.381080:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572965.381083:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572965.381087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572965.381099:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864004224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572965.381106:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572965.381109:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572965.381110:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572965.381113:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4715 00000100:00080000:2.0:1788572965.381116:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572965.381119:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788572965.381122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572965.381130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864004352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572965.381137:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572965.381140:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572965.381141:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572965.381144:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4715 00000100:00080000:2.0:1788572965.381146:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572965.381156:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788572965.381158:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788572965.381161:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572965.381174:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864004480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572965.381207:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff92fc4617ed80 x1875449864003968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572965.381223:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864003968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572965.381231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572965.381234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572965.381237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864003968:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572965.381259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004096:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572965.381276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572965.381292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff92fc4617c700 x1875449864004096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572965.381300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864004096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572965.381306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572965.381308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572965.381310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004096:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572965.381314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004224:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572965.381326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572965.381332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff92fc4617f100 x1875449864004224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572965.381339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864004224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572965.381344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572965.381346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572965.381348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004224:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572965.381351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004352:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572965.381362:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572965.381368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff92fc4617f800 x1875449864004352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572965.381375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864004352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572965.381380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572965.381382:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572965.381384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004352:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572965.381388:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004480:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572965.381398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572965.381404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572965/real 1788572965] req@ffff92fc4617f480 x1875449864004480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572965.381410:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864004480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572965.381415:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572965.381417:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572965.381419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004480:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572965.828886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572965.892828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572966.852949:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572966.916789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572967.876873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572967.940826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572968.900875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572968.964811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572969.924831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572969.988805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572970.500916:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572970.500924:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572970.500928:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572970.500937:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4721 00000100:00080000:3.0:1788572970.500946:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572970.500953:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572970.501000:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864004608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572970.501026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572970.501029:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572970.501030:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572970.501033:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4721 00000100:00080000:3.0:1788572970.501037:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572970.501041:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572970.501051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572970.501066:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864004736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572970.501078:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572970.501081:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864004608:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572970.501083:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572970.501088:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572970.501097:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4721 00000100:00080000:3.0:1788572970.501102:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572970.501106:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572970.501112:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572970.501127:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864004864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572970.501141:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572970.501143:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572970.501144:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572970.501148:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4721 00000100:00080000:3.0:1788572970.501151:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572970.501153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572970.501157:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572970.501162:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572970.501171:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864004992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572970.501179:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572970.501183:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572970.501185:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572970.501189:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4721 00000100:00080000:3.0:1788572970.501193:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572970.501197:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572970.501203:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572970.501215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff92fc6733aa00 x1875449864004608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572970.501217:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864005120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572970.501261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864004608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572970.501275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572970.501279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572970.501284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864004608:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572970.501309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004736:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572970.501335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572970.501349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff92fc67338e00 x1875449864004736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572970.501363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864004736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572970.501372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572970.501376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572970.501380:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004736:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572970.501387:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004864:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572970.501409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572970.501421:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff92fc6733a680 x1875449864004864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572970.501442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864004864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572970.501452:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572970.501455:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572970.501459:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004864:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572970.501465:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004992:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572970.501487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572970.501499:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff92fc67338700 x1875449864004992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572970.501511:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864004992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572970.501519:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572970.501522:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572970.501526:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864004992:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572970.501532:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005120:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572970.501552:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572970.501561:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572970/real 1788572970] req@ffff92fc6733b800 x1875449864005120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572970.501578:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864005120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572970.501588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572970.501591:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572970.501596:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005120:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572970.948853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572971.012833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572971.972857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572972.036799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572972.997387:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572973.060854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572974.020880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572974.084790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572975.044852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572975.108816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572975.620842:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.620851:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572975.620854:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.620863:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4726 00000100:00080000:3.0:1788572975.620872:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.620879:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572975.620920:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864005248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572975.620941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.620944:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572975.620946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.620948:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4726 00000100:00080000:3.0:1788572975.620951:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.620953:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572975.620962:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572975.620988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864005376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572975.620998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864005248:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572975.621003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.621008:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572975.621011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.621016:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4726 00000100:00080000:3.0:1788572975.621020:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.621025:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572975.621031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572975.621045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864005504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572975.621052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.621055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572975.621056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.621058:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4726 00000100:00080000:3.0:1788572975.621062:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572975.621066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572975.621069:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572975.621074:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572975.621083:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864005632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572975.621089:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.621092:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572975.621093:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.621095:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4726 00000100:00080000:3.0:1788572975.621097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.621099:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572975.621102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572975.621111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864005760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572975.621119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff92fc6733b480 x1875449864005248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572975.621143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864005248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572975.621155:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572975.621158:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572975.621161:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864005248:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572975.621180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005376:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572975.621200:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572975.621209:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff92fc67339500 x1875449864005376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572975.621218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864005376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572975.621223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572975.621225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572975.621229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005376:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572975.621235:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005504:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572975.621255:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572975.621266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff92fc67338000 x1875449864005504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572975.621279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864005504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572975.621288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572975.621292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572975.621295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005504:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572975.621302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005632:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572975.621321:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572975.621333:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff92fc6733bb80 x1875449864005632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572975.621344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864005632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572975.621352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572975.621356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572975.621359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005632:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572975.621365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005760:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572975.621383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572975.621394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff92fc6733a300 x1875449864005760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572975.621415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864005760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572975.621425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572975.621428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572975.621430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864005760:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572976.068884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572976.132807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572977.092880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572977.156832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572978.116883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572978.180780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572979.140849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572979.204823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572980.164869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572980.228843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572980.740816:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572980.740827:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572980.740832:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572980.740843:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4731 00000100:00080000:3.0:1788572980.740854:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572980.740863:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572980.740902:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864005888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572980.740935:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572980.740941:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572980.740943:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572980.740961:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4731 00000100:00080000:3.0:1788572980.740965:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572980.740968:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572980.740977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572980.740982:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864005888:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788572980.740992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864006016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572980.741001:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572980.741004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572980.741005:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572980.741008:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4731 00000100:00080000:3.0:1788572980.741011:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572980.741013:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572980.741016:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572980.741030:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864006144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572980.741036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572980.741038:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572980.741040:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572980.741042:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4731 00000100:00080000:3.0:1788572980.741046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572980.741048:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572980.741051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572980.741057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788572980.741061:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864006272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572980.741068:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572980.741071:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572980.741072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572980.741075:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4731 00000100:00080000:3.0:1788572980.741078:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572980.741081:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572980.741083:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572980.741090:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864006400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572980.741101:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff92fc67338380 x1875449864005888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572980.741117:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864005888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572980.741126:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572980.741129:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572980.741132:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864005888:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572980.741150:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006016:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572980.741167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572980.741184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff92fc6733b100 x1875449864006016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572980.741196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864006016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572980.741202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572980.741205:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572980.741207:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006016:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572980.741211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006144:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572980.741223:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572980.741230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff92fc67338a80 x1875449864006144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572980.741243:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864006144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572980.741252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572980.741255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572980.741258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006144:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572980.741264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006272:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572980.741279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572980.741287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff92fc67339880 x1875449864006272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572980.741295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864006272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572980.741300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572980.741302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572980.741304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006272:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572980.741308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006400:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572980.741319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572980.741324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff92fc67339c00 x1875449864006400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572980.741331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864006400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572980.741341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572980.741343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572980.741345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006400:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572981.188817:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572981.252832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572982.212961:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572982.276855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572983.236837:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572983.300890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572984.260906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572984.324850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572985.284847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572985.348848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572985.860845:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572985.860855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572985.860860:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572985.860871:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4736 00000100:00080000:3.0:1788572985.860881:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572985.860888:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572985.860962:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864006528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572985.860996:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572985.861008:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572985.861010:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572985.861015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4736 00000100:00080000:3.0:1788572985.861020:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572985.861024:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572985.861035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572985.861051:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864006656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572985.861055:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864006528:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572985.861072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572985.861076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572985.861078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572985.861082:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4736 00000100:00080000:3.0:1788572985.861086:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572985.861090:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572985.861095:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572985.861108:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864006784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572985.861118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572985.861122:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572985.861124:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572985.861127:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4736 00000100:00080000:3.0:1788572985.861132:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572985.861136:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788572985.861141:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572985.861148:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572985.861163:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864006912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572985.861172:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572985.861176:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572985.861178:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572985.861184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff92fc6733ad80 x1875449864006528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572985.861185:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4736 00000100:00080000:3.0:1788572985.861191:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572985.861198:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572985.861206:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572985.861216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864006528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572985.861219:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864007040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572985.861242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572985.861248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572985.861252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864006528:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572985.861276:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006656:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572985.861302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572985.861311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff92fc67339180 x1875449864006656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572985.861320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864006656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572985.861326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572985.861328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572985.861331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006656:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572985.861334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006784:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572985.861345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572985.861351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff92fc67339f80 x1875449864006784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572985.861358:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864006784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572985.861363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572985.861365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572985.861367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006784:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572985.861370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006912:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572985.861381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572985.861386:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff92fc67339c00 x1875449864006912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572985.861393:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864006912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572985.861400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572985.861403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572985.861407:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864006912:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572985.861412:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007040:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572985.861431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572985.861442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff92fc67339880 x1875449864007040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572985.861455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864007040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572985.861461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572985.861463:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572985.861465:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007040:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572986.308883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572986.372845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572987.332872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572987.396842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572988.356913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572988.420866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572989.380907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572989.444859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572990.404936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572990.468894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788572990.980857:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572990.980869:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572990.980874:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572990.980885:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4741 00000100:00080000:3.0:1788572990.980896:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572990.980905:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788572990.980964:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864007168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572990.980984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572990.980988:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572990.980989:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572990.980993:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4741 00000100:00080000:3.0:1788572990.980997:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572990.981000:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572990.981011:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572990.981022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864007296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572990.981031:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864007168:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788572990.981033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572990.981040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572990.981043:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572990.981048:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4741 00000100:00080000:3.0:1788572990.981052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572990.981055:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572990.981059:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572990.981067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864007424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572990.981074:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572990.981076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572990.981078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572990.981082:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4741 00000100:00080000:3.0:1788572990.981085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572990.981088:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788572990.981092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572990.981102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864007552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572990.981115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572990.981118:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572990.981119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572990.981122:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4741 00000100:00080000:3.0:1788572990.981126:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572990.981130:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788572990.981134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788572990.981136:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572990.981146:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864007680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572990.981191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff92fc67338a80 x1875449864007168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572990.981211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864007168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572990.981220:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572990.981224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572990.981227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864007168:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788572990.981253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007296:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572990.981281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572990.981299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff92fc6733b100 x1875449864007296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572990.981315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864007296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572990.981326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572990.981329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572990.981332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007296:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572990.981338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007424:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788572990.981355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572990.981364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff92fc67338380 x1875449864007424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572990.981378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864007424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572990.981384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572990.981386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572990.981388:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007424:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788572990.981392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007552:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572990.981411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572990.981419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff92fc6733a300 x1875449864007552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572990.981429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864007552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572990.981434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572990.981436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572990.981439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007552:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788572990.981443:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007680:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788572990.981455:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788572990.981463:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff92fc6733bb80 x1875449864007680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572990.981472:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864007680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572990.981483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572990.981486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788572990.981490:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007680:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572991.428977:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572991.492859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572992.452928:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572992.516928:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572993.476867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572993.540796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572994.500931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572994.564884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572995.524924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572995.588783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788572996.100873:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572996.100883:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572996.100888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572996.100911:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4746 00000100:00080000:0.0:1788572996.100922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572996.100931:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788572996.100969:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864007808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572996.100998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572996.101001:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572996.101002:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572996.101005:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4746 00000100:00080000:0.0:1788572996.101009:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572996.101013:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572996.101021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572996.101035:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864007936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572996.101047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572996.101052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864007808:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788572996.101053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572996.101058:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572996.101064:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4746 00000100:00080000:0.0:1788572996.101070:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572996.101074:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572996.101079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572996.101094:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864008064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572996.101104:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572996.101108:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572996.101110:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572996.101114:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4746 00000100:00080000:0.0:1788572996.101118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572996.101122:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572996.101129:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572996.101141:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864008192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572996.101158:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572996.101162:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572996.101164:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572996.101169:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4746 00000100:00100000:1.0:1788572996.101172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007936:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788572996.101175:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572996.101182:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788572996.101188:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572996.101204:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864008320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572996.101211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008064:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572996.101245:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008192:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572996.101271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008320:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788572996.548851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572996.612827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572997.572860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572997.636856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788572998.596913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572998.660861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788572999.172993:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:3.0:1788572999.173031:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572999.173045:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572999.173086:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572999.173098:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572999.173107:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788572999.173115:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788572999.173143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572996/real 1788572999] req@ffff92fc67049c00 x1875449864008320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572999.173169:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864008320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572999.173181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572999.173186:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572999.173191:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008320:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572999.173198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572996/real 1788572999] req@ffff92fc67048e00 x1875449864007936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572999.173211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864007936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572999.173226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572999.173230:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572999.173234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864007936:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572999.173241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572996/real 1788572999] req@ffff92fc6704bb80 x1875449864008064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572999.173253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864008064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572999.173261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572999.173264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572999.173268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008064:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788572999.173274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572996/real 1788572999] req@ffff92fc67049180 x1875449864008192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572999.173285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864008192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572999.173293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572999.173296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572999.173299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008192:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788572999.173307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572996/real 1788572999] req@ffff92fc67049500 x1875449864007808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572999.173318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864007808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572999.173326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572999.173330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788572999.173333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864007808:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788572999.620850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572999.684827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573000.644913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573000.708869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573001.220817:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573001.220829:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573001.220834:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573001.220858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4751 00000100:00080000:0.0:1788573001.220869:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573001.220907:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573001.220944:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449864008448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573001.220978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573001.220982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573001.220984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573001.220988:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4751 00000100:00080000:0.0:1788573001.220993:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573001.220997:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573001.221008:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573001.221021:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449864008576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573001.221043:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864008448:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573001.221044:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573001.221052:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573001.221056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573001.221061:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4751 00000100:00080000:0.0:1788573001.221064:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573001.221069:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573001.221074:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573001.221089:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864008704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573001.221098:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573001.221102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573001.221103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573001.221107:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4751 00000100:00080000:0.0:1788573001.221121:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573001.221125:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573001.221131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788573001.221132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573001.221151:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864008832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573001.221161:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573001.221164:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573001.221167:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573001.221170:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4751 00000100:00080000:0.0:1788573001.221174:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573001.221179:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573001.221185:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573001.221202:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864008960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573001.221202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff92fc6704b800 x1875449864008448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573001.221236:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449864008448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573001.221252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573001.221257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573001.221262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864008448:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573001.221296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008576:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573001.221327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573001.221345:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff92fc6704b480 x1875449864008576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573001.221357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449864008576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573001.221367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573001.221371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573001.221374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008576:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573001.221381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008704:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573001.221401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573001.221413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff92fc6704b100 x1875449864008704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573001.221426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864008704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573001.221433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573001.221437:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573001.221440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008704:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573001.221445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008832:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573001.221467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573001.221478:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff92fc6704ad80 x1875449864008832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573001.221490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864008832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573001.221499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573001.221502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573001.221506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008832:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573001.221524:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008960:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573001.221545:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573001.221557:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573001/real 1788573001] req@ffff92fc6704a680 x1875449864008960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573001.221570:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864008960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573001.221579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573001.221582:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573001.221585:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864008960:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573001.668864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573001.732830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573002.692880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573002.756822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573003.716886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573003.780828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573004.740896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573004.804818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573005.764906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573005.828874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573006.340810:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573006.340818:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573006.340822:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573006.340830:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4756 00000100:00080000:3.0:1788573006.340865:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573006.340872:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573006.340903:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864009088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573006.340927:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573006.340931:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573006.340932:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573006.340935:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4756 00000100:00080000:3.0:1788573006.340938:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573006.340940:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573006.340948:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573006.340957:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864009216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573006.340964:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573006.340966:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573006.340968:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573006.340970:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4756 00000100:00080000:3.0:1788573006.340972:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573006.340974:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573006.340978:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573006.340984:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864009088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573006.340987:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864009344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573006.340996:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573006.340999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573006.341000:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573006.341003:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4756 00000100:00080000:3.0:1788573006.341005:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573006.341007:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573006.341011:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573006.341019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864009472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573006.341026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573006.341028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573006.341030:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573006.341033:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4756 00000100:00080000:3.0:1788573006.341036:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573006.341038:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573006.341041:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573006.341047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864009600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573006.341056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573006.341092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff92fc67338000 x1875449864009088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573006.341106:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864009088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573006.341121:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573006.341124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573006.341127:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864009088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573006.341132:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009216:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573006.341148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573006.341157:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff92fc67339500 x1875449864009216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573006.341165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864009216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573006.341174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573006.341177:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573006.341179:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009216:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573006.341197:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009344:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573006.341208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573006.341215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff92fc6733b480 x1875449864009344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573006.341222:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864009344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573006.341228:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573006.341230:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573006.341232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009344:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573006.341235:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009472:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573006.341245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573006.341251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff92fc6733b800 x1875449864009472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573006.341258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864009472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573006.341263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573006.341265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573006.341268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009472:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573006.341271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009600:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573006.341281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573006.341286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573006/real 1788573006] req@ffff92fc67338700 x1875449864009600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573006.341293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864009600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573006.341298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573006.341300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573006.341302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009600:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573006.788887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573006.852878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573007.812888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573007.876870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573008.836963:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573008.900822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573009.860836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573009.924834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573010.884934:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573010.948826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573011.460865:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573011.460885:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573011.460890:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573011.460901:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4762 00000100:00080000:3.0:1788573011.460914:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573011.460924:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573011.460976:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864009728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573011.461012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573011.461020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573011.461022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573011.461027:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4762 00000100:00080000:3.0:1788573011.461032:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573011.461035:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573011.461047:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573011.461066:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864009856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573011.461080:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864009728:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573011.461083:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573011.461095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573011.461098:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573011.461103:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4762 00000100:00080000:3.0:1788573011.461106:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573011.461108:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573011.461111:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573011.461126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864009984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573011.461133:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573011.461136:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573011.461138:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573011.461141:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4762 00000100:00080000:3.0:1788573011.461143:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573011.461145:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573011.461150:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573011.461159:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864010112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573011.461167:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573011.461170:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573011.461171:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573011.461174:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4762 00000100:00080000:3.0:1788573011.461177:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573011.461183:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573011.461185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573011.461194:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573011.461211:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864010240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573011.461243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff92fc6733a680 x1875449864009728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573011.461259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864009728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573011.461269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573011.461273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573011.461276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864009728:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573011.461300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009856:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573011.461319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573011.461330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff92fc67338e00 x1875449864009856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573011.461338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864009856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573011.461345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573011.461347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573011.461350:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009856:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573011.461354:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009984:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573011.461365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573011.461372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff92fc6733aa00 x1875449864009984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573011.461380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864009984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573011.461385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573011.461387:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573011.461390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864009984:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573011.461394:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010112:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573011.461405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573011.461411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff92fc67338700 x1875449864010112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573011.461418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864010112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573011.461423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573011.461426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573011.461428:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010112:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573011.461431:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010240:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573011.461441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573011.461447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573011/real 1788573011] req@ffff92fc6733b800 x1875449864010240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573011.461454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864010240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573011.461461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573011.461465:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573011.461468:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010240:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573011.908963:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573011.972849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573012.932813:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573012.996827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573013.956857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573014.020807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573014.980892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573015.044773:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573016.004844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573016.068796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573016.580831:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.580843:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573016.580847:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.580858:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4767 00000100:00080000:3.0:1788573016.580867:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.580874:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573016.580910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864010368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.580940:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.580944:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573016.580947:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.580950:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4767 00000100:00080000:3.0:1788573016.580954:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.580958:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573016.580967:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573016.580980:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864010496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.580992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573016.580999:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864010368:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573016.580999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573016.581002:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.581010:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4767 00000100:00080000:3.0:1788573016.581014:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.581018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573016.581022:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573016.581034:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864010624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.581044:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.581047:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573016.581049:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.581053:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4767 00000100:00080000:3.0:1788573016.581057:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.581061:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573016.581067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573016.581074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573016.581079:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864010752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573016.581097:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573016.581100:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573016.581102:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573016.581107:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4767 00000100:00080000:3.0:1788573016.581111:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573016.581115:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573016.581120:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff92fc6733b480 x1875449864010368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573016.581122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573016.581137:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864010880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573016.581148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864010368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.581173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.581178:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573016.581182:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864010368:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573016.581208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010496:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573016.581233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573016.581246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff92fc67339500 x1875449864010496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.581257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864010496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.581265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.581269:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573016.581272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010496:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573016.581278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010624:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573016.581298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573016.581310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff92fc67338000 x1875449864010624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.581321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864010624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.581329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.581332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573016.581336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010624:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573016.581342:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010752:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573016.581362:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573016.581373:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff92fc6733bb80 x1875449864010752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.581393:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864010752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.581401:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.581403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573016.581405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010752:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573016.581410:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010880:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573016.581424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573016.581431:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff92fc6733a300 x1875449864010880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.581439:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864010880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.581444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.581446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573016.581448:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864010880:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573017.028875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573017.092838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573018.052870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573018.116874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573019.076912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573019.140883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573020.100868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573020.166002:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573021.124850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573021.188847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573021.700878:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.700903:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573021.700914:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.700936:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4772 00000100:00080000:3.0:1788573021.700944:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.700951:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573021.700982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864011008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.701006:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.701009:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573021.701011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.701014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4772 00000100:00080000:3.0:1788573021.701016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.701018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573021.701027:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573021.701038:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864011136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.701047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.701049:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573021.701051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573021.701054:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864011008:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573021.701056:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4772 00000100:00080000:3.0:1788573021.701061:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.701064:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573021.701067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573021.701075:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864011264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.701082:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.701084:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573021.701086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573021.701088:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4772 00000100:00080000:3.0:1788573021.701090:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.701093:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573021.701096:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573021.701103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864011392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573021.701109:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573021.701111:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573021.701113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573021.701117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573021.701122:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4772 00000100:00080000:3.0:1788573021.701126:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573021.701129:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573021.701133:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573021.701142:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864011520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573021.701160:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff92fc67338380 x1875449864011008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.701177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864011008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.701196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.701200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573021.701205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864011008:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573021.701229:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011136:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573021.701247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573021.701256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff92fc6733b100 x1875449864011136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.701264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864011136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.701270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.701273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573021.701275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011136:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573021.701279:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011264:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573021.701290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573021.701296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff92fc67338a80 x1875449864011264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.701303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864011264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.701308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.701310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573021.701312:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011264:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573021.701316:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011392:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573021.701327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573021.701333:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff92fc67339880 x1875449864011392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.701340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864011392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.701345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.701347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573021.701348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011392:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573021.701352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011520:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573021.701362:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573021.701367:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff92fc67339c00 x1875449864011520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.701374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864011520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.701379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.701380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573021.701382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011520:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573022.148805:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573022.212838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573023.172936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573023.236857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573024.196829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573024.260803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573025.220841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573025.284805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573026.244836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573026.308839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573026.820829:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573026.820841:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573026.820846:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573026.820857:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4777 00000100:00080000:3.0:1788573026.820869:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573026.820877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573026.820908:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864011648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573026.820932:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573026.820935:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573026.820937:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573026.820940:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4777 00000100:00080000:3.0:1788573026.820943:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573026.820945:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573026.820953:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573026.820967:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864011776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573026.820980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573026.820983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573026.820985:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573026.820987:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4777 00000100:00080000:3.0:1788573026.820990:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573026.820992:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573026.820997:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573026.821001:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864011648:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573026.821007:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864011904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573026.821015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573026.821017:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573026.821018:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573026.821020:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4777 00000100:00080000:3.0:1788573026.821022:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573026.821024:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573026.821027:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573026.821036:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864012032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573026.821042:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573026.821045:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573026.821046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573026.821049:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4777 00000100:00080000:3.0:1788573026.821052:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573026.821054:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573026.821057:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573026.821071:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864012160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573026.821073:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573026.821115:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff92fc67339f80 x1875449864011648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.821131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864011648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.821140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.821143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573026.821146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864011648:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573026.821151:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011776:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573026.821170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573026.821182:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff92fc67339180 x1875449864011776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.821192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864011776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.821199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.821202:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573026.821205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011776:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573026.821227:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011904:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573026.821243:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573026.821262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff92fc6733ad80 x1875449864011904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.821280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864011904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.821296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.821302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573026.821306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864011904:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573026.821314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012032:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573026.821344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573026.821356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff92fc67339c00 x1875449864012032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.821367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864012032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.821373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.821376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573026.821379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012032:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573026.821385:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012160:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573026.821402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573026.821411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff92fc67339880 x1875449864012160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.821419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864012160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.821425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.821427:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573026.821431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012160:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573027.268873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573027.332835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573028.292821:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573028.356795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573029.316837:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573029.380834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573030.340847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573030.404815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573031.364828:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573031.428827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573031.940796:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573031.940805:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573031.940808:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573031.940822:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4782 00000100:00080000:3.0:1788573031.940833:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573031.940842:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573031.940879:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864012288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573031.940906:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573031.940911:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573031.940913:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573031.940917:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4782 00000100:00080000:3.0:1788573031.940921:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573031.940938:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573031.940948:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864012288:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573031.940953:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573031.940968:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864012416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573031.940982:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573031.940987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573031.940989:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573031.940991:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4782 00000100:00080000:3.0:1788573031.940995:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573031.940998:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573031.941002:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573031.941009:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573031.941012:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864012544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573031.941020:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573031.941022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573031.941024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573031.941027:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4782 00000100:00080000:3.0:1788573031.941030:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573031.941033:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573031.941036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573031.941043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864012672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573031.941049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573031.941051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573031.941052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573031.941055:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4782 00000100:00080000:3.0:1788573031.941058:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573031.941061:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573031.941064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573031.941071:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff92fc67338a80 x1875449864012288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573031.941073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864012800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573031.941117:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864012288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.941134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.941138:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573031.941141:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864012288:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573031.941160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012416:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573031.941189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573031.941198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff92fc6733b100 x1875449864012416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.941208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864012416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.941219:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.941222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573031.941225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012416:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573031.941229:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012544:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573031.941242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573031.941249:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff92fc67338380 x1875449864012544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.941260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864012544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.941270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.941273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573031.941277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012544:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573031.941284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012672:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573031.941305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573031.941316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff92fc6733a300 x1875449864012672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.941326:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864012672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.941331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.941334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573031.941336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012672:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573031.941341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012800:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573031.941356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573031.941363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff92fc6733bb80 x1875449864012800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.941387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864012800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.941393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.941397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573031.941400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864012800:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573032.388857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573032.452777:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573033.412885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573033.476814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573034.436858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573034.500769:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573035.460825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573035.524807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573036.484852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573036.548826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573037.060863:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573037.060881:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573037.060887:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573037.060900:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4787 00000100:00080000:3.0:1788573037.060915:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573037.060926:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573037.060979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864012928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573037.061015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573037.061019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573037.061022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573037.061027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4787 00000100:00080000:3.0:1788573037.061031:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573037.061035:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573037.061049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573037.061070:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864013056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573037.061084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573037.061091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573037.061092:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864012928:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573037.061095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573037.061114:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4787 00000100:00080000:3.0:1788573037.061121:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573037.061128:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573037.061136:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573037.061152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864013184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573037.061160:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573037.061162:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573037.061164:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573037.061168:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4787 00000100:00080000:3.0:1788573037.061171:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573037.061174:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573037.061178:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573037.061185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573037.061192:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864013312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573037.061199:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573037.061202:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573037.061204:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573037.061207:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4787 00000100:00080000:3.0:1788573037.061211:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573037.061214:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573037.061217:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573037.061224:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864013440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573037.061251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573037/real 1788573037] req@ffff92fc67338000 x1875449864012928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573037.061308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864012928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573037.061322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573037.061326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573037.061331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864012928:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573037.061360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013056:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573037.061390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573037.061404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573037/real 1788573037] req@ffff92fc67339500 x1875449864013056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573037.061413:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864013056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573037.061420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573037.061422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573037.061426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013056:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573037.061433:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013184:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573037.061455:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573037.061468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573037/real 1788573037] req@ffff92fc6733b480 x1875449864013184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573037.061482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864013184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573037.061490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573037.061494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573037.061497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013184:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573037.061504:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013312:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573037.061528:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573037.061541:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573037/real 1788573037] req@ffff92fc6733b800 x1875449864013312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573037.061551:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864013312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573037.061558:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573037.061561:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573037.061563:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013312:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573037.061568:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013440:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573037.061583:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573037.061590:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573037/real 1788573037] req@ffff92fc67338700 x1875449864013440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573037.061598:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864013440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573037.061603:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573037.061605:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573037.061607:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013440:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573037.508913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573037.572865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573038.532864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573038.596844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573039.556838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573039.620904:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573040.580913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573040.644883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573041.604807:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573041.668825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573042.180823:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573042.180832:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573042.180836:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573042.180845:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4792 00000100:00080000:3.0:1788573042.180854:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573042.180861:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573042.180895:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864013568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573042.180918:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573042.180921:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573042.180922:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573042.180925:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4792 00000100:00080000:3.0:1788573042.180927:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573042.180930:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573042.180938:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573042.180946:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864013696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573042.180954:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573042.180956:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573042.180958:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573042.180960:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4792 00000100:00080000:3.0:1788573042.180962:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573042.180967:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573042.180969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864013568:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573042.180971:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573042.180979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864013824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573042.180987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573042.180989:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573042.180991:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573042.180993:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4792 00000100:00080000:3.0:1788573042.180995:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573042.180997:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573042.181001:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573042.181010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864013952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573042.181016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573042.181018:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573042.181020:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573042.181022:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4792 00000100:00080000:3.0:1788573042.181024:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573042.181026:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573042.181029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573042.181030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573042.181038:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864014080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573042.181070:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff92fc6733aa00 x1875449864013568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573042.181091:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864013568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573042.181102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573042.181106:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573042.181111:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864013568:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573042.181128:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013696:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573042.181149:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573042.181159:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff92fc67338e00 x1875449864013696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573042.181167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864013696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573042.181192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573042.181195:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573042.181197:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013696:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573042.181216:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013824:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573042.181231:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573042.181239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff92fc6733a680 x1875449864013824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573042.181247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864013824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573042.181252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573042.181254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573042.181256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013824:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573042.181260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013952:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573042.181271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573042.181277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff92fc67338700 x1875449864013952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573042.181285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864013952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573042.181289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573042.181291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573042.181294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864013952:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573042.181298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014080:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573042.181308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573042.181313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573042/real 1788573042] req@ffff92fc6733b800 x1875449864014080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573042.181320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864014080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573042.181325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573042.181327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573042.181329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014080:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573042.628870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573042.692831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573043.652954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573043.716826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573044.676862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573044.740833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573045.700886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573045.764812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573046.724894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573046.788855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573047.300857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573047.300869:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573047.300875:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573047.300886:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4797 00000100:00080000:3.0:1788573047.300898:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573047.300906:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573047.300950:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864014208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573047.300980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573047.300984:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573047.300985:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573047.300988:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4797 00000100:00080000:3.0:1788573047.300991:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573047.300993:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573047.301003:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573047.301012:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864014336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573047.301020:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573047.301023:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573047.301025:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573047.301028:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4797 00000100:00080000:3.0:1788573047.301030:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573047.301032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573047.301036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573047.301043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864014464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573047.301056:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573047.301059:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864014208:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573047.301060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573047.301063:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573047.301070:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4797 00000100:00080000:3.0:1788573047.301073:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573047.301075:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573047.301079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573047.301118:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864014592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573047.301126:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573047.301129:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573047.301131:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573047.301134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4797 00000100:00080000:3.0:1788573047.301137:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573047.301140:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573047.301143:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573047.301152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864014720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573047.301164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573047.301229:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff92fc6733b480 x1875449864014208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573047.301247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864014208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573047.301256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573047.301259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573047.301262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864014208:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573047.301268:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014336:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573047.301287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573047.301298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff92fc67339500 x1875449864014336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573047.301306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864014336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573047.301312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573047.301314:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573047.301316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014336:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573047.301338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014464:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573047.301349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573047.301356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff92fc67338000 x1875449864014464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573047.301363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864014464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573047.301368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573047.301370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573047.301372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014464:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573047.301376:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014592:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573047.301387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573047.301392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff92fc6733bb80 x1875449864014592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573047.301400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864014592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573047.301406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573047.301408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573047.301410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014592:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573047.301413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014720:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573047.301422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573047.301428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573047/real 1788573047] req@ffff92fc6733a300 x1875449864014720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573047.301434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864014720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573047.301438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573047.301441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573047.301443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014720:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573047.748910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573047.812825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573048.772884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573048.836866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573049.796849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573049.860890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573050.820860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573050.884880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573051.844908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573051.908880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573052.420833:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573052.420844:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573052.420848:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573052.420859:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4803 00000100:00080000:3.0:1788573052.420870:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573052.420878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573052.420916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864014848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573052.420950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573052.420953:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573052.420956:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573052.420960:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4803 00000100:00080000:3.0:1788573052.420963:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573052.420967:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573052.420977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573052.421004:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864014976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573052.421015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573052.421016:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864014848:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573052.421020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573052.421024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573052.421029:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4803 00000100:00080000:3.0:1788573052.421032:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573052.421034:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573052.421039:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573052.421050:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864015104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573052.421057:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573052.421060:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573052.421061:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573052.421063:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4803 00000100:00080000:3.0:1788573052.421065:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573052.421067:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573052.421070:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573052.421077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864015232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573052.421082:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573052.421084:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573052.421085:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573052.421090:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4803 00000800:00000100:1.0:1788573052.421091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573052.421095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573052.421099:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573052.421102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573052.421111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864015360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573052.421133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff92fc67338380 x1875449864014848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573052.421149:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864014848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573052.421158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573052.421161:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573052.421164:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864014848:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573052.421182:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014976:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573052.421199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573052.421207:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff92fc6733b100 x1875449864014976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573052.421216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864014976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573052.421221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573052.421224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573052.421226:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864014976:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573052.421230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015104:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573052.421240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573052.421247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff92fc67338a80 x1875449864015104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573052.421254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864015104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573052.421259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573052.421261:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573052.421263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015104:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573052.421266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015232:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573052.421277:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573052.421282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff92fc67339880 x1875449864015232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573052.421290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864015232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573052.421295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573052.421297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573052.421299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015232:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573052.421303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015360:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573052.421312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573052.421317:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573052/real 1788573052] req@ffff92fc67339c00 x1875449864015360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573052.421324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864015360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573052.421329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573052.421331:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573052.421333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015360:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573052.868877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573052.932854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573053.892846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573053.956803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573054.916867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573054.980821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573055.940832:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573056.004837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573056.964853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573057.028874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573057.540891:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.540939:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573057.540944:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.540956:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4808 00000100:00080000:2.0:1788573057.540968:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.540977:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573057.541030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864015488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.541056:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.541060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573057.541061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.541064:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4808 00000100:00080000:2.0:1788573057.541067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.541070:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573057.541080:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573057.541090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864015616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.541098:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.541101:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573057.541103:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.541106:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4808 00000100:00080000:2.0:1788573057.541108:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.541112:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573057.541116:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573057.541127:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864015488:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573057.541127:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864015744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.541144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.541147:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573057.541148:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.541151:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4808 00000100:00080000:2.0:1788573057.541153:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.541155:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573057.541159:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573057.541168:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864015872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573057.541175:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573057.541178:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573057.541179:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573057.541182:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4808 00000100:00080000:2.0:1788573057.541185:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573057.541187:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573057.541190:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573057.541197:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864016000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573057.541210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573057.541258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff92fc4617ca80 x1875449864015488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573057.541272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864015488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573057.541287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573057.541291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573057.541294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864015488:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573057.541299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015616:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573057.541316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573057.541325:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff92fc4617dc00 x1875449864015616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573057.541334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864015616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573057.541340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573057.541342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573057.541345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573057.541373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015744:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573057.541385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573057.541392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff92fc4617e680 x1875449864015744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573057.541400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864015744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573057.541406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573057.541408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573057.541411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573057.541415:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015872:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573057.541432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573057.541443:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff92fc4617d180 x1875449864015872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573057.541457:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864015872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573057.541467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573057.541471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573057.541475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864015872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573057.541481:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016000:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573057.541498:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573057.541506:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573057/real 1788573057] req@ffff92fc4617c380 x1875449864016000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573057.541514:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864016000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573057.541520:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573057.541522:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573057.541525:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016000:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573057.988883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573058.052926:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573059.012857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573059.076845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573060.036872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573060.100826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573061.060822:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573061.124824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573062.084881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573062.148821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573062.660918:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573062.660929:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573062.660934:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573062.660946:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4813 00000100:00080000:3.0:1788573062.660959:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573062.660968:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573062.661019:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864016128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573062.661059:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573062.661063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573062.661065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573062.661070:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4813 00000100:00080000:3.0:1788573062.661074:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573062.661078:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573062.661090:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573062.661105:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864016256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573062.661122:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864016128:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573062.661128:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573062.661134:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573062.661137:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573062.661141:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4813 00000100:00080000:3.0:1788573062.661144:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573062.661147:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573062.661152:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573062.661163:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864016384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573062.661241:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573062.661245:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788573062.661247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016256:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573062.661249:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573062.661260:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4813 00000100:00080000:3.0:1788573062.661266:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573062.661271:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573062.661277:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573062.661288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016384:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573062.661292:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864016512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573062.661446:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573062.661449:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573062.661450:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573062.661454:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4813 00000100:00080000:3.0:1788573062.661457:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573062.661461:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573062.661470:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573062.661481:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864016640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573062.661504:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016512:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573062.661550:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016640:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573063.108880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573063.172875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573064.132863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573064.196841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573065.156921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573065.220893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573065.733046:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573065.733092:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573065.733109:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573065.733149:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573065.733159:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573065.733170:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573065.733180:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573065.733200:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff92fc67339c00 x1875449864016512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573065.733229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864016512/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573065.733242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573065.733249:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573065.733254:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016512:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573065.733263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff92fc67339880 x1875449864016640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573065.733277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864016640/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573065.733286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573065.733290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573065.733293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016640:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573065.733299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff92fc67339180 x1875449864016256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573065.733312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864016256/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573065.733321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573065.733325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573065.733328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016256:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573065.733335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff92fc67339f80 x1875449864016384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573065.733348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864016384/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573065.733356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573065.733359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573065.733362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016384:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573065.733368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573065] req@ffff92fc6733ad80 x1875449864016128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573065.733380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864016128/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573065.733387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573065.733390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573065.733393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864016128:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573066.180863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573066.244854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573067.204862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573067.268807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573067.780839:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573067.780851:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573067.780855:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573067.780864:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4818 00000100:00080000:3.0:1788573067.780873:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573067.780879:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573067.780910:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864016768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573067.780934:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573067.780939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573067.780940:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573067.780943:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4818 00000100:00080000:3.0:1788573067.780945:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573067.780948:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573067.780958:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573067.780979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864016896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573067.780990:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864016768:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573067.780994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573067.781002:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573067.781004:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573067.781009:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4818 00000100:00080000:3.0:1788573067.781012:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573067.781017:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573067.781022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573067.781036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864017024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573067.781047:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573067.781050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573067.781052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573067.781057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4818 00000100:00080000:3.0:1788573067.781061:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573067.781065:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573067.781071:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573067.781073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573067.781087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864017152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573067.781098:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573067.781102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573067.781103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573067.781107:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4818 00000100:00080000:3.0:1788573067.781111:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573067.781116:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573067.781121:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573067.781127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff92fc67338a80 x1875449864016768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573067.781135:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864017280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573067.781165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864016768/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573067.781178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573067.781183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573067.781186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864016768:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573067.781212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016896:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573067.781240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573067.781254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff92fc6733b100 x1875449864016896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573067.781267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864016896/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573067.781277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573067.781280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573067.781284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864016896:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573067.781291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017024:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573067.781314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573067.781327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff92fc67338380 x1875449864017024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573067.781341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864017024/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573067.781356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573067.781360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573067.781363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017024:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573067.781370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017152:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573067.781393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573067.781406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff92fc6733a300 x1875449864017152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573067.781420:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864017152/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573067.781428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573067.781432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573067.781435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017152:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573067.781441:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017280:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573067.781462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573067.781474:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff92fc6733bb80 x1875449864017280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573067.781486:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864017280/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573067.781495:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573067.781498:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573067.781502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017280:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573068.228906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573068.292887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573069.252844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573069.316790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573070.276862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573070.340823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573071.300873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573071.364804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573072.324809:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573072.388809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573072.900819:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.900827:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573072.900831:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.900847:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4823 00000100:00080000:3.0:1788573072.900857:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.900862:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573072.900898:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864017408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.900921:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.900924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573072.900925:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.900928:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4823 00000100:00080000:3.0:1788573072.900931:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.900934:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573072.900941:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573072.900952:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864017536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.900960:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.900962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573072.900964:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.900966:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4823 00000100:00080000:3.0:1788573072.900968:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.900973:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573072.900975:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864017408:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573072.900977:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573072.900987:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864017664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.900995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.900997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573072.900998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.901000:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4823 00000100:00080000:3.0:1788573072.901002:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.901004:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573072.901008:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573072.901016:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864017792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.901026:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.901029:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573072.901030:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.901033:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4823 00000800:00000100:1.0:1788573072.901040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573072.901042:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.901047:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573072.901051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573072.901058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864017920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573072.901081:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff92fc67338000 x1875449864017408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573072.901097:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864017408/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573072.901105:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573072.901128:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573072.901132:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864017408:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573072.901137:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017536:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573072.901153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573072.901162:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff92fc67339500 x1875449864017536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573072.901170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864017536/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573072.901176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573072.901178:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573072.901180:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017536:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573072.901199:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017664:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573072.901210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573072.901217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff92fc6733b480 x1875449864017664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573072.901224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864017664/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573072.901230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573072.901231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573072.901234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017664:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573072.901237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017792:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573072.901246:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573072.901252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff92fc6733b800 x1875449864017792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573072.901259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864017792/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573072.901263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573072.901265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573072.901267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017792:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573072.901271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017920:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573072.901280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573072.901285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff92fc67338700 x1875449864017920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573072.901292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864017920/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573072.901297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573072.901299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573072.901301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864017920:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573073.348854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573073.412833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573074.372863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573074.436807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573075.396854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573075.460843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573076.420861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573076.484884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573077.444940:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573077.508824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573078.020801:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573078.020810:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573078.020813:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573078.020823:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4828 00000100:00080000:3.0:1788573078.020832:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573078.020838:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573078.020873:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864018048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573078.020899:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573078.020902:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573078.020904:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573078.020906:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4828 00000100:00080000:3.0:1788573078.020909:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573078.020911:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573078.020920:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573078.020930:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864018176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573078.020943:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573078.020947:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573078.020951:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864018048:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573078.020951:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573078.020961:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4828 00000100:00080000:3.0:1788573078.020965:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573078.020969:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573078.020975:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573078.020988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864018304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573078.020995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573078.020997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573078.020998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573078.021001:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4828 00000100:00080000:3.0:1788573078.021004:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573078.021007:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573078.021011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573078.021015:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573078.021021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864018432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573078.021028:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573078.021031:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573078.021032:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573078.021036:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4828 00000100:00080000:3.0:1788573078.021038:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573078.021040:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573078.021043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573078.021051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864018560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573078.021058:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573078/real 1788573078] req@ffff92fc6733a680 x1875449864018048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573078.021074:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864018048/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573078.021083:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573078.021087:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573078.021090:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864018048:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573078.021115:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018176:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573078.021140:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573078.021153:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573078/real 1788573078] req@ffff92fc67338e00 x1875449864018176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573078.021161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864018176/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573078.021167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573078.021170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573078.021172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018176:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573078.021177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018304:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573078.021191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573078.021198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573078/real 1788573078] req@ffff92fc6733aa00 x1875449864018304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573078.021205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864018304/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573078.021210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573078.021212:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573078.021213:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018304:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573078.021217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018432:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573078.021228:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573078.021234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573078/real 1788573078] req@ffff92fc67338700 x1875449864018432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573078.021241:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864018432/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573078.021246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573078.021248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573078.021250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018432:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573078.021253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018560:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573078.021263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573078.021268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573078/real 1788573078] req@ffff92fc6733b800 x1875449864018560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573078.021276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864018560/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573078.021282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573078.021283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573078.021285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018560:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573078.468906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573078.532828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573079.492903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573079.556842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573080.516887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573080.580815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573081.540895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573081.604865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573082.564875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573082.628812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573083.140880:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573083.140894:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573083.140901:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573083.140915:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4833 00000100:00080000:3.0:1788573083.140928:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573083.140939:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573083.140998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864018688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573083.141026:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573083.141029:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573083.141031:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573083.141034:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4833 00000100:00080000:3.0:1788573083.141037:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573083.141040:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573083.141050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573083.141060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864018816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573083.141074:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573083.141077:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573083.141080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573083.141083:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864018688:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573083.141085:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4833 00000100:00080000:3.0:1788573083.141090:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573083.141095:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573083.141099:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573083.141105:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864018944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573083.141112:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573083.141115:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573083.141116:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573083.141119:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4833 00000100:00080000:3.0:1788573083.141121:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573083.141123:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573083.141126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573083.141136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864019072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573083.141142:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573083.141144:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573083.141145:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573083.141148:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4833 00000100:00080000:3.0:1788573083.141168:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573083.141172:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573083.141176:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573083.141184:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864019200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573083.141210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573083.141268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573083/real 1788573083] req@ffff92fc6733b480 x1875449864018688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573083.141290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864018688/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573083.141303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573083.141308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573083.141313:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864018688:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573083.141343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018816:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573083.141370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573083.141385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573083/real 1788573083] req@ffff92fc67339500 x1875449864018816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573083.141398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864018816/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573083.141413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573083.141417:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573083.141421:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018816:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573083.141428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018944:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573083.141450:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573083.141462:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573083/real 1788573083] req@ffff92fc67338000 x1875449864018944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573083.141476:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864018944/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573083.141484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573083.141487:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573083.141491:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864018944:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573083.141498:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019072:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573083.141518:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573083.141529:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573083/real 1788573083] req@ffff92fc6733bb80 x1875449864019072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573083.141542:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864019072/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573083.141550:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573083.141554:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573083.141558:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019072:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573083.141564:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019200:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573083.141580:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573083.141589:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573083/real 1788573083] req@ffff92fc6733a300 x1875449864019200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573083.141602:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864019200/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573083.141610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573083.141613:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573083.141617:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019200:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573083.588857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573083.652842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573084.612851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573084.676795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573085.636898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573085.700842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573086.660916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573086.724813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573087.684936:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573087.748877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573088.260846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573088.260855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573088.260858:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573088.260867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4838 00000100:00080000:0.0:1788573088.260876:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573088.260882:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573088.260916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864019328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573088.260942:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573088.260951:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573088.260964:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573088.260968:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4838 00000100:00080000:0.0:1788573088.260971:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573088.260973:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573088.260981:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573088.260991:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864019456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573088.260999:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573088.261001:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573088.261003:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573088.261006:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4838 00000100:00080000:0.0:1788573088.261009:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573088.261015:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573088.261015:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864019328:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788573088.261022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573088.261029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864019584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573088.261035:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573088.261037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573088.261039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573088.261042:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4838 00000100:00080000:0.0:1788573088.261044:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573088.261046:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573088.261049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573088.261056:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864019712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573088.261062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573088.261064:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573088.261066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573088.261068:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4838 00000100:00080000:0.0:1788573088.261085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573088.261087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573088.261090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573088.261096:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573088.261100:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864019840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573088.261140:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff92fc6704aa00 x1875449864019328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573088.261155:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864019328/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573088.261162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573088.261166:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573088.261169:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864019328:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573088.261174:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019456:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573088.261190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573088.261199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff92fc6704a300 x1875449864019456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573088.261207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864019456/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573088.261212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573088.261214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573088.261217:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019456:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573088.261234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019584:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573088.261245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573088.261251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff92fc67049880 x1875449864019584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573088.261259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864019584/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573088.261272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573088.261274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573088.261276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019584:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573088.261281:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019712:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573088.261292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573088.261298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff92fc67048000 x1875449864019712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573088.261306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864019712/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573088.261312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573088.261314:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573088.261316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019712:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573088.261319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019840:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573088.261329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573088.261335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573088/real 1788573088] req@ffff92fc67049f80 x1875449864019840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573088.261342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864019840/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573088.261347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573088.261349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573088.261351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864019840:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573088.708874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573088.772786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573089.732872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573089.796834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573090.756855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573090.820872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573091.780865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573091.844859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573092.804872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573092.868876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573093.380850:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573093.380861:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573093.380866:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573093.380876:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4843 00000100:00080000:2.0:1788573093.380888:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573093.380895:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573093.380940:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864019968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573093.380968:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573093.380971:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573093.380973:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573093.380976:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4843 00000100:00080000:2.0:1788573093.380978:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573093.380980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573093.380991:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573093.381000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864020096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573093.381007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573093.381010:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573093.381011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573093.381014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4843 00000100:00080000:2.0:1788573093.381016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573093.381018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573093.381021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573093.381026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864019968:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573093.381030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864020224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573093.381040:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573093.381043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573093.381044:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573093.381046:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4843 00000100:00080000:2.0:1788573093.381048:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573093.381051:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573093.381054:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573093.381061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864020352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573093.381067:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573093.381071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573093.381072:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573093.381075:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4843 00000100:00080000:2.0:1788573093.381078:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573093.381080:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573093.381083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573093.381091:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864020480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573093.381100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573093.381145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff92fc4617fb80 x1875449864019968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573093.381161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864019968/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573093.381170:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573093.381174:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573093.381177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864019968:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573093.381184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020096:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573093.381199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573093.381208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff92fc4617c000 x1875449864020096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573093.381216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864020096/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573093.381222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573093.381224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573093.381226:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020096:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573093.381256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020224:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573093.381275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573093.381287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff92fc4617d880 x1875449864020224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573093.381300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864020224/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573093.381309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573093.381312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573093.381315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020224:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573093.381319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020352:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573093.381336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573093.381343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff92fc4617e300 x1875449864020352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573093.381352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864020352/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573093.381357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573093.381359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573093.381361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020352:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573093.381365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020480:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573093.381375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573093.381381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573093/real 1788573093] req@ffff92fc4617ea00 x1875449864020480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573093.381388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864020480/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573093.381393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573093.381395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573093.381397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020480:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573093.828907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573093.892860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573094.852867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573094.916836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573095.876836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573095.940808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573096.900848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573096.964782:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573097.924823:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573097.988822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573098.500832:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573098.500841:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573098.500846:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573098.500855:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4849 00000100:00080000:2.0:1788573098.500864:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573098.500871:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573098.500903:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864020608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573098.500927:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573098.500930:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573098.500931:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573098.500934:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4849 00000100:00080000:2.0:1788573098.500936:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573098.500939:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573098.500947:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573098.500960:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864020736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573098.500968:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573098.500971:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573098.500972:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573098.500975:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4849 00000100:00080000:2.0:1788573098.500978:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573098.500980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573098.500984:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573098.500988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864020608:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573098.500995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864020864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573098.501005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573098.501009:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573098.501010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573098.501012:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4849 00000100:00080000:2.0:1788573098.501014:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573098.501017:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573098.501021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573098.501030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864020992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573098.501036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573098.501038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573098.501039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573098.501042:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4849 00000100:00080000:2.0:1788573098.501045:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573098.501047:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573098.501050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573098.501058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864021120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573098.501064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573098.501107:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff92fc4617d500 x1875449864020608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573098.501126:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864020608/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573098.501144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573098.501148:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573098.501152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864020608:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573098.501160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020736:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573098.501183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573098.501196:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff92fc4617ce00 x1875449864020736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573098.501210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864020736/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573098.501218:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573098.501221:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573098.501224:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020736:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573098.501249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020864:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573098.501271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573098.501283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff92fc4617ea00 x1875449864020864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573098.501294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864020864/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573098.501302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573098.501305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573098.501308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020864:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573098.501315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020992:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573098.501335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573098.501347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff92fc4617e300 x1875449864020992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573098.501361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864020992/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573098.501371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573098.501376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573098.501381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864020992:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573098.501390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021120:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573098.501413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573098.501432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573098/real 1788573098] req@ffff92fc4617d880 x1875449864021120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573098.501445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864021120/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573098.501454:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573098.501457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573098.501460:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021120:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573098.948846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573099.012809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573099.972839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573100.036824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573100.996839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573101.060843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573102.020849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573102.084808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573103.044874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573103.108848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573103.620884:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573103.620893:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573103.620896:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573103.620905:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4854 00000100:00080000:0.0:1788573103.620914:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573103.620920:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573103.620950:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864021248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573103.620977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573103.620982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573103.620984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573103.620988:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4854 00000100:00080000:0.0:1788573103.620991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573103.620996:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573103.621005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573103.621018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864021376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573103.621026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864021248:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573103.621029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573103.621032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573103.621034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573103.621038:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4854 00000100:00080000:0.0:1788573103.621041:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573103.621044:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573103.621049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573103.621058:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864021504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573103.621066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573103.621069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573103.621071:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573103.621074:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4854 00000100:00080000:0.0:1788573103.621078:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573103.621081:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573103.621085:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573103.621089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573103.621095:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864021632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573103.621103:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573103.621106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573103.621108:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573103.621111:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4854 00000100:00080000:0.0:1788573103.621113:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573103.621117:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573103.621121:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573103.621127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff92fc67048a80 x1875449864021248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573103.621130:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864021760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573103.621160:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864021248/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573103.621179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573103.621184:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573103.621189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864021248:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573103.621230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021376:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573103.621250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573103.621260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff92fc67048380 x1875449864021376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573103.621269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864021376/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573103.621275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573103.621277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573103.621279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021376:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573103.621283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021504:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573103.621295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573103.621301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff92fc67048700 x1875449864021504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573103.621308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864021504/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573103.621313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573103.621315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573103.621317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021504:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573103.621321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021632:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573103.621332:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573103.621338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff92fc67049f80 x1875449864021632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573103.621345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864021632/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573103.621350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573103.621351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573103.621361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021632:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573103.621392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021760:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573103.621407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573103.621414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff92fc67048000 x1875449864021760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573103.621421:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864021760/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573103.621426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573103.621428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573103.621430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864021760:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573104.068883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573104.132813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573105.092898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573105.156850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573106.116858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573106.180806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573107.140899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573107.204835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573108.164956:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573108.228828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573108.740876:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573108.740885:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573108.740889:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573108.740898:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4859 00000100:00080000:3.0:1788573108.740907:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573108.740913:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573108.740953:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864021888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573108.740984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573108.740987:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573108.740988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573108.740991:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4859 00000100:00080000:3.0:1788573108.740995:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573108.740998:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573108.741006:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573108.741018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864022016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573108.741026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573108.741030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573108.741032:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573108.741036:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4859 00000100:00100000:1.0:1788573108.741039:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864021888:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573108.741040:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573108.741043:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573108.741049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573108.741056:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864022144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573108.741064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573108.741066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573108.741067:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573108.741070:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4859 00000100:00080000:3.0:1788573108.741071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573108.741074:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573108.741077:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573108.741084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864022272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573108.741089:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573108.741092:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573108.741093:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573108.741095:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4859 00000100:00080000:3.0:1788573108.741098:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573108.741100:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573108.741104:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573108.741115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864022400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573108.741116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573108.741161:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff92fc67338380 x1875449864021888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573108.741199:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864021888/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573108.741210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573108.741214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573108.741219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864021888:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573108.741226:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022016:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573108.741248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573108.741261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff92fc6733b100 x1875449864022016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573108.741274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864022016/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573108.741283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573108.741286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573108.741290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022016:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573108.741312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022144:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573108.741342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573108.741355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff92fc67338a80 x1875449864022144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573108.741368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864022144/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573108.741382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573108.741385:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573108.741388:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022144:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573108.741395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022272:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573108.741413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573108.741424:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff92fc6733ad80 x1875449864022272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573108.741434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864022272/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573108.741443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573108.741446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573108.741449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022272:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573108.741455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022400:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573108.741471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573108.741481:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff92fc67339f80 x1875449864022400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573108.741492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864022400/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573108.741500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573108.741503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573108.741507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022400:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573109.188831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573109.252849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573110.212889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573110.276793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573111.236820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573111.300834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573112.260854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573112.324826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573113.284867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573113.348842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573113.860910:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573113.860921:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573113.860926:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573113.860937:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4864 00000100:00080000:2.0:1788573113.860949:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573113.860957:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573113.861035:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864022528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573113.861072:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573113.861077:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573113.861079:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573113.861084:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4864 00000100:00080000:2.0:1788573113.861088:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573113.861094:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573113.861114:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573113.861123:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864022528:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573113.861134:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864022656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573113.861149:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573113.861153:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573113.861155:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573113.861160:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4864 00000100:00080000:2.0:1788573113.861165:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573113.861170:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573113.861176:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573113.861191:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864022784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573113.861195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573113.861208:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573113.861213:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573113.861215:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573113.861220:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4864 00000100:00080000:2.0:1788573113.861226:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573113.861231:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573113.861237:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573113.861245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff92fc4617c000 x1875449864022528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573113.861255:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864022912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573113.861267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864022528/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573113.861268:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573113.861280:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573113.861284:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.861289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573113.861291:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4864 00000100:00080000:2.0:1788573113.861298:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.861299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788573113.861306:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573113.861309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864022528:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788573113.861315:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573113.861331:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864023040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573113.861348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022656:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573113.861371:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573113.861381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff92fc4617fb80 x1875449864022656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573113.861390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864022656/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573113.861397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573113.861399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573113.861402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022656:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573113.861406:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022784:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573113.861417:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573113.861425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff92fc4617c380 x1875449864022784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573113.861433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864022784/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573113.861439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573113.861441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573113.861444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022784:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573113.861447:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022912:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573113.861459:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573113.861474:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff92fc4617d180 x1875449864022912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573113.861483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864022912/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573113.861488:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573113.861490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573113.861493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864022912:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573113.861496:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023040:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573113.861508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573113.861514:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff92fc4617e680 x1875449864023040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573113.861521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864023040/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573113.861527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573113.861529:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573113.861531:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023040:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573114.308893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573114.372915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573115.332944:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573115.396863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573116.356861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573116.420821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573117.380817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573117.444844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573118.404868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573118.468852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573118.980796:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.980806:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573118.980811:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.980822:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4869 00000100:00080000:2.0:1788573118.980833:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.980842:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573118.980886:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864023168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.980915:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.980921:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573118.980922:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.980925:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4869 00000100:00080000:2.0:1788573118.980928:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.980930:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573118.980941:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573118.980950:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864023296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.980959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.980962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573118.980964:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.980967:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4869 00000100:00080000:2.0:1788573118.980970:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573118.980975:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864023168:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573118.980975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573118.980980:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573118.980991:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864023424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.981001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.981005:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573118.981008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.981013:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4869 00000100:00080000:2.0:1788573118.981017:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.981022:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573118.981030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573118.981044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864023552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.981051:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.981054:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788573118.981058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573118.981058:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.981065:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4869 00000100:00080000:2.0:1788573118.981069:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.981072:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573118.981077:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573118.981086:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864023680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573118.981111:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff92fc4617dc00 x1875449864023168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573118.981130:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864023168/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573118.981140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573118.981143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573118.981147:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864023168:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573118.981153:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023296:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573118.981170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573118.981179:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff92fc4617ca80 x1875449864023296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573118.981187:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864023296/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573118.981193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573118.981195:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573118.981198:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023296:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573118.981225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023424:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573118.981240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573118.981251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff92fc4617f480 x1875449864023424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573118.981264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864023424/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573118.981274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573118.981277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573118.981281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023424:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573118.981288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023552:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573118.981307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573118.981315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff92fc4617f800 x1875449864023552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573118.981323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864023552/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573118.981329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573118.981338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573118.981341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023552:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573118.981345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023680:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573118.981358:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573118.981365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff92fc4617f100 x1875449864023680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573118.981373:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864023680/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573118.981379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573118.981380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573118.981382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023680:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573119.428905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573119.492822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573120.452870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573120.516837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573121.476866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573121.540807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573122.500836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573122.564848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573123.524845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573123.588785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573124.100871:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573124.100883:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573124.100888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573124.100901:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4874 00000100:00080000:0.0:1788573124.100915:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573124.100925:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573124.100983:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864023808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573124.101008:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573124.101012:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573124.101014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573124.101017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4874 00000100:00080000:0.0:1788573124.101020:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573124.101022:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573124.101032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573124.101040:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864023936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573124.101049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573124.101051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573124.101053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573124.101055:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4874 00000100:00080000:0.0:1788573124.101057:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573124.101060:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573124.101063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573124.101073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864024064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573124.101079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573124.101082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573124.101083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573124.101086:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4874 00000100:00080000:0.0:1788573124.101087:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573124.101090:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573124.101094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573124.101103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864024192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573124.101117:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573124.101117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864023808:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573124.101121:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573124.101126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573124.101135:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4874 00000100:00080000:0.0:1788573124.101139:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573124.101142:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573124.101147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573124.101157:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864024320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573124.101218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573124.101266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573124/real 1788573124] req@ffff92fc67049880 x1875449864023808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573124.101286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864023808/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573124.101298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573124.101302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573124.101307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864023808:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573124.101315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023936:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573124.101341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573124.101355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573124/real 1788573124] req@ffff92fc6704a300 x1875449864023936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573124.101368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864023936/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573124.101384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573124.101387:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573124.101391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864023936:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573124.101398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024064:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573124.101419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573124.101432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573124/real 1788573124] req@ffff92fc6704aa00 x1875449864024064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573124.101445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864024064/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573124.101453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573124.101457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573124.101461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024064:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573124.101500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024192:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573124.101524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573124.101536:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573124/real 1788573124] req@ffff92fc6704a680 x1875449864024192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573124.101550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864024192/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573124.101559:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573124.101563:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573124.101567:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024192:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573124.101574:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024320:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573124.101611:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573124.101623:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573124/real 1788573124] req@ffff92fc6704ad80 x1875449864024320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573124.101636:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864024320/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573124.101654:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573124.101658:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573124.101661:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024320:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573124.548835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573124.612826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573125.572894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573125.636811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573126.596886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573126.660846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573127.620939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573127.684828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573128.644937:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573128.708861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573129.220911:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573129.220919:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573129.220922:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573129.220932:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4879 00000100:00080000:2.0:1788573129.220943:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573129.220950:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573129.220989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864024448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573129.221016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573129.221020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573129.221021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573129.221024:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4879 00000100:00080000:2.0:1788573129.221028:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573129.221031:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573129.221040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573129.221050:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864024576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573129.221059:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864024448:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573129.221060:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573129.221069:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573129.221070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573129.221074:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4879 00000100:00080000:2.0:1788573129.221076:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573129.221079:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573129.221082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573129.221091:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864024704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573129.221098:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573129.221100:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573129.221102:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573129.221104:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4879 00000100:00080000:2.0:1788573129.221107:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573129.221109:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573129.221112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573129.221119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864024832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573129.221130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573129.221133:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573129.221135:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573129.221139:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4879 00000100:00080000:2.0:1788573129.221145:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573129.221150:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573129.221157:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573129.221171:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024576:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573129.221175:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864024960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573129.221204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024704:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573129.221223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024832:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573129.221239:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024960:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573129.668920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573129.732883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573130.692924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573130.756852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573131.716860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573131.780811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573132.293000:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573132.293047:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573132.293065:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573132.293104:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573132.293114:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573132.293122:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573132.293134:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573132.293162:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff92fc4617f480 x1875449864024960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573132.293191:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864024960/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573132.293211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573132.293218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573132.293222:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024960:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573132.293231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff92fc4617ed80 x1875449864024576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573132.293245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864024576/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573132.293254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573132.293258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573132.293273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024576:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573132.293282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff92fc4617f100 x1875449864024704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573132.293295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864024704/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573132.293303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573132.293307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573132.293311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024704:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573132.293319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff92fc4617f800 x1875449864024832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573132.293331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864024832/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573132.293340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573132.293344:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573132.293348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864024832:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573132.293355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573129/real 1788573132] req@ffff92fc4617c700 x1875449864024448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573132.293385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864024448/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573132.293395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573132.293399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573132.293403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864024448:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573132.740819:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573132.804805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573133.764867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573133.828835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573134.340799:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.340810:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573134.340813:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.340825:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4884 00000100:00080000:2.0:1788573134.340837:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.340846:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573134.340886:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864025088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573134.340917:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.340922:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573134.340925:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.340930:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4884 00000100:00080000:2.0:1788573134.340935:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.340940:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573134.340951:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573134.340959:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864025088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573134.340971:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864025216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573134.340985:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.340988:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573134.340990:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.340993:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4884 00000100:00080000:2.0:1788573134.340996:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.341000:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573134.341004:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573134.341016:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864025344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573134.341026:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573134.341030:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573134.341033:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573134.341036:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573134.341041:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4884 00000100:00080000:2.0:1788573134.341046:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.341051:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573134.341057:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573134.341071:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864025472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573134.341080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573134.341086:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff92fc4617ca80 x1875449864025088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573134.341086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573134.341090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573134.341100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4884 00000100:00080000:2.0:1788573134.341107:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573134.341112:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573134.341118:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864025088/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573134.341119:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573134.341133:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864025600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573134.341159:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573134.341168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573134.341173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864025088:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573134.341219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025216:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573134.341247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573134.341261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff92fc4617dc00 x1875449864025216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573134.341273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864025216/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573134.341281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573134.341285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573134.341289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025216:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573134.341295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025344:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573134.341316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573134.341330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff92fc4617e680 x1875449864025344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573134.341356:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864025344/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573134.341365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573134.341368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573134.341371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025344:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573134.341378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025472:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573134.341399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573134.341411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff92fc4617d180 x1875449864025472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573134.341421:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864025472/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573134.341429:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573134.341432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573134.341435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025472:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573134.341447:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025600:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573134.341466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573134.341477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573134/real 1788573134] req@ffff92fc4617c380 x1875449864025600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573134.341495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864025600/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573134.341503:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573134.341506:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573134.341510:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025600:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573134.788836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573134.852833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573135.812958:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573135.876870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573136.836915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573136.900841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573137.860903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573137.924862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573138.884923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573138.948829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573139.460834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573139.460845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573139.460849:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573139.460860:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4890 00000100:00080000:2.0:1788573139.460869:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573139.460876:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573139.460910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864025728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573139.460935:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573139.460939:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573139.460940:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573139.460943:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4890 00000100:00080000:2.0:1788573139.460945:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573139.460947:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573139.460954:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573139.460964:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864025856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573139.460971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573139.460974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573139.460975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573139.460978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4890 00000100:00080000:2.0:1788573139.460979:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573139.460984:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573139.460986:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864025728:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788573139.460988:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573139.460998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864025984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573139.461017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573139.461020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573139.461021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573139.461024:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4890 00000100:00080000:2.0:1788573139.461027:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573139.461030:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573139.461034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573139.461044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864026112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573139.461050:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573139.461053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573139.461054:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573139.461057:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4890 00000100:00080000:2.0:1788573139.461059:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573139.461061:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573139.461065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573139.461067:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573139.461076:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864026240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573139.461106:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff92fc4617fb80 x1875449864025728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573139.461120:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864025728/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573139.461128:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573139.461131:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573139.461133:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864025728:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573139.461139:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025856:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573139.461155:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573139.461163:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff92fc4617c000 x1875449864025856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573139.461173:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864025856/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573139.461182:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573139.461186:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573139.461189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025856:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573139.461215:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025984:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573139.461235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573139.461247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff92fc4617d880 x1875449864025984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573139.461259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864025984/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573139.461267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573139.461270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573139.461273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864025984:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573139.461278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026112:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573139.461296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573139.461305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff92fc4617e300 x1875449864026112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573139.461321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864026112/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573139.461329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573139.461332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573139.461336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026112:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573139.461341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026240:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573139.461357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573139.461366:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573139/real 1788573139] req@ffff92fc4617ea00 x1875449864026240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573139.461377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864026240/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573139.461385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573139.461388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573139.461391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026240:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573139.908876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573139.972796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573140.932922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573140.996824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573141.956849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573142.020826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573142.980849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573143.044832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573144.004881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573144.068825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573144.580907:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573144.580918:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573144.580923:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573144.580933:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4895 00000100:00080000:3.0:1788573144.580945:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573144.580953:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573144.580997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864026368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573144.581024:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573144.581027:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573144.581028:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573144.581031:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4895 00000100:00080000:3.0:1788573144.581033:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573144.581036:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573144.581046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573144.581055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864026496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573144.581064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573144.581067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573144.581068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573144.581071:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4895 00000100:00080000:3.0:1788573144.581074:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573144.581077:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573144.581080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573144.581084:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864026368:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573144.581091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864026624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573144.581099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573144.581102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573144.581102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573144.581105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4895 00000100:00080000:3.0:1788573144.581107:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573144.581110:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573144.581113:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573144.581124:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864026752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573144.581130:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573144.581132:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573144.581133:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573144.581136:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4895 00000100:00080000:3.0:1788573144.581139:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573144.581141:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573144.581145:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573144.581153:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864026880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573144.581159:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573144.581211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff92fc67339180 x1875449864026368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573144.581229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864026368/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573144.581238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573144.581242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573144.581245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864026368:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573144.581251:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026496:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573144.581267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573144.581276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff92fc67339880 x1875449864026496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573144.581285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864026496/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573144.581293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573144.581297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573144.581301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026496:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573144.581340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026624:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573144.581359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573144.581368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff92fc67339c00 x1875449864026624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573144.581381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864026624/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573144.581387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573144.581390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573144.581392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026624:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573144.581395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026752:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573144.581407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573144.581413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff92fc67339f80 x1875449864026752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573144.581421:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864026752/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573144.581426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573144.581428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573144.581430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026752:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573144.581433:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026880:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573144.581452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573144.581459:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573144] req@ffff92fc6733ad80 x1875449864026880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573144.581467:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864026880/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573144.581477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573144.581479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573144.581482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864026880:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788573145.028963:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573145.092843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573146.052896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573146.116833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573147.076917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573147.140797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573148.100847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573148.164796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573149.124926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573149.188786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573149.700850:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573149.700858:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573149.700862:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573149.700871:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4900 00000100:00080000:3.0:1788573149.700880:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573149.700886:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573149.700924:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864027008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573149.700946:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573149.700949:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573149.700951:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573149.700953:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4900 00000100:00080000:3.0:1788573149.700956:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573149.700958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573149.700966:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573149.700974:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864027136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573149.700981:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573149.700984:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573149.700986:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573149.700989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4900 00000100:00080000:3.0:1788573149.700991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573149.700993:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573149.700997:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573149.701007:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864027008:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573149.701011:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864027264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573149.701019:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573149.701021:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573149.701022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573149.701025:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4900 00000100:00080000:3.0:1788573149.701029:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573149.701031:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573149.701035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573149.701045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864027392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573149.701051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573149.701054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573149.701055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573149.701057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4900 00000100:00080000:3.0:1788573149.701059:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573149.701061:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573149.701064:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573149.701071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864027520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573149.701077:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573149.701117:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff92fc67338a80 x1875449864027008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.701130:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864027008/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.701137:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.701140:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573149.701143:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864027008:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573149.701148:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027136:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573149.701163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573149.701171:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff92fc6733b100 x1875449864027136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.701184:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864027136/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.701189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.701191:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573149.701193:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027136:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573149.701211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027264:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573149.701222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573149.701229:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff92fc67338380 x1875449864027264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.701236:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864027264/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.701242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.701244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573149.701246:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027264:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573149.701249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027392:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573149.701259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573149.701264:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff92fc6733a300 x1875449864027392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.701271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864027392/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.701281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.701283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573149.701285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027392:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573149.701288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027520:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573149.701298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573149.701304:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff92fc6733bb80 x1875449864027520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.701310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864027520/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.701315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.701317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573149.701319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027520:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573150.148867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573150.212859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573151.172954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573151.236856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573152.196849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573152.260806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573153.220901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573153.284817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573154.244866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573154.308845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573154.820813:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573154.820821:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573154.820826:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573154.820835:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4905 00000100:00080000:3.0:1788573154.820843:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573154.820849:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573154.820880:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864027648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573154.820904:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573154.820907:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573154.820913:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573154.820916:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4905 00000100:00080000:3.0:1788573154.820918:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573154.820921:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573154.820929:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573154.820939:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864027776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573154.820947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573154.820950:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573154.820951:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573154.820954:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4905 00000100:00080000:3.0:1788573154.820956:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573154.820958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573154.820962:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573154.820965:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864027648:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573154.820976:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864027904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573154.820996:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573154.821000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573154.821002:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573154.821007:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4905 00000100:00080000:3.0:1788573154.821012:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573154.821016:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573154.821022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573154.821033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573154.821034:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864028032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573154.821043:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573154.821046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573154.821048:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573154.821051:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4905 00000100:00080000:3.0:1788573154.821056:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573154.821060:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573154.821065:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573154.821079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864028160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573154.821081:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff92fc67338000 x1875449864027648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.821101:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864027648/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.821112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.821116:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573154.821118:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864027648:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573154.821124:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027776:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573154.821143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573154.821151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff92fc67339500 x1875449864027776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.821159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864027776/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.821165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.821167:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573154.821169:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027776:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573154.821188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027904:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573154.821198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573154.821204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff92fc6733b480 x1875449864027904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.821211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864027904/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.821221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.821223:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573154.821225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864027904:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573154.821228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028032:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573154.821238:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573154.821244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff92fc6733b800 x1875449864028032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.821251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864028032/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.821256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.821257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573154.821259:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028032:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573154.821262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028160:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573154.821272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573154.821277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff92fc67338700 x1875449864028160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.821284:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864028160/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.821289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.821291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573154.821293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028160:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788573155.268924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573155.332818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573156.292804:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573156.356812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573157.316926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573157.380831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573158.340996:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573158.404874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573159.364902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573159.428876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573159.940900:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.940911:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573159.940916:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.940927:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4910 00000100:00080000:2.0:1788573159.940938:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.940945:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573159.940983:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864028288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.941014:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.941017:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573159.941019:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.941022:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4910 00000100:00080000:2.0:1788573159.941026:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.941029:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573159.941037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573159.941047:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864028416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.941055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.941057:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573159.941059:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.941062:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4910 00000100:00080000:2.0:1788573159.941064:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.941077:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573159.941087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573159.941098:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864028544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.941107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.941110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573159.941111:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.941114:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4910 00000100:00080000:2.0:1788573159.941117:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.941125:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573159.941127:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864028288:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788573159.941130:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573159.941152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864028672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573159.941163:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573159.941167:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573159.941169:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573159.941174:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4910 00000100:00080000:2.0:1788573159.941179:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573159.941183:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573159.941187:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573159.941202:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864028800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573159.941242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573159.941330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff92fc4617ce00 x1875449864028288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.941350:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864028288/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.941360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.941364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573159.941367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864028288:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573159.941374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028416:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573159.941394:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573159.941404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff92fc4617d500 x1875449864028416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.941423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864028416/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.941430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.941433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573159.941436:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028416:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573159.941463:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028544:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573159.941479:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573159.941487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff92fc4617ea00 x1875449864028544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.941496:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864028544/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.941502:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.941505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573159.941507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028544:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573159.941511:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028672:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573159.941523:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573159.941530:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff92fc4617e300 x1875449864028672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.941538:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864028672/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.941544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.941546:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573159.941549:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028672:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573159.941553:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028800:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573159.941565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573159.941572:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff92fc4617d880 x1875449864028800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.941581:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864028800/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.941586:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.941588:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573159.941590:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864028800:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573160.388901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573160.452829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573161.412931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573161.476830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573162.436901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573162.500810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573163.460910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573163.524844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573164.484914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573164.548858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573165.060893:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573165.060908:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573165.060914:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573165.060927:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4915 00000100:00080000:3.0:1788573165.060942:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573165.060953:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573165.061006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864028928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573165.061041:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573165.061046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573165.061049:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573165.061053:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4915 00000100:00080000:3.0:1788573165.061058:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573165.061062:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573165.061075:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573165.061091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864029056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573165.061105:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573165.061106:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864028928:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573165.061125:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573165.061130:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573165.061135:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4915 00000100:00080000:3.0:1788573165.061140:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573165.061144:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573165.061150:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573165.061165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864029184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573165.061189:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573165.061194:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573165.061196:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573165.061200:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4915 00000800:00000100:0.0:1788573165.061206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573165.061207:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573165.061215:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573165.061222:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573165.061238:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864029312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573165.061249:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573165.061253:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573165.061255:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573165.061260:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4915 00000100:00080000:3.0:1788573165.061265:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573165.061268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573165/real 1788573165] req@ffff92fc6733aa00 x1875449864028928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573165.061273:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573165.061281:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573165.061297:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864029440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573165.061301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864028928/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573165.061325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573165.061332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573165.061339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864028928:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573165.061372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029056:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573165.061399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573165.061414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573165/real 1788573165] req@ffff92fc67338e00 x1875449864029056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573165.061429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864029056/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573165.061439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573165.061442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573165.061446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029056:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573165.061464:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029184:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573165.061487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573165.061500:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573165/real 1788573165] req@ffff92fc6733a680 x1875449864029184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573165.061513:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864029184/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573165.061522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573165.061526:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573165.061530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029184:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573165.061536:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029312:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573165.061557:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573165.061570:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573165/real 1788573165] req@ffff92fc67338700 x1875449864029312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573165.061583:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864029312/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573165.061591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573165.061595:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573165.061599:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029312:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573165.061605:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029440:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573165.061627:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573165.061639:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573165/real 1788573165] req@ffff92fc6733b800 x1875449864029440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573165.061653:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864029440/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573165.061661:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573165.061665:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573165.061668:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029440:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788573165.508930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573165.572899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573166.532867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573166.596795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573167.556877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573167.620800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573168.580864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573168.644843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573169.604900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573169.668915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573170.180866:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573170.180882:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573170.180888:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573170.180904:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4920 00000100:00080000:3.0:1788573170.180919:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573170.180931:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573170.180993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864029568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573170.181031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573170.181036:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573170.181038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573170.181042:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4920 00000100:00080000:3.0:1788573170.181047:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573170.181052:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573170.181065:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573170.181089:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864029696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573170.181101:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864029568:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573170.181105:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573170.181113:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573170.181117:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573170.181123:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4920 00000100:00080000:3.0:1788573170.181128:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573170.181135:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573170.181141:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573170.181181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573170.181229:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff92fc6733b480 x1875449864029568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573170.181236:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864029824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573170.181251:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573170.181256:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573170.181258:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573170.181261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864029568/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573170.181267:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4920 00000100:00080000:0.0:1788573170.181277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573170.181277:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573170.181287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788573170.181287:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573170.181297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864029568:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573170.181299:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573170.181317:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864029952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573170.181330:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573170.181334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029696:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573170.181336:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788573170.181366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573170.181377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff92fc67339500 x1875449864029696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573170.181387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864029696/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573170.181393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573170.181396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573170.181398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029696:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573170.181403:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029824:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573170.181416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573170.181423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff92fc67338000 x1875449864029824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573170.181437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864029824/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573170.181443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573170.181445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573170.181448:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029824:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573170.181459:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029952:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573170.181479:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573170.181490:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573170.181495:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff92fc6733bb80 x1875449864029952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573170.181500:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4920 00000100:00080000:3.0:1788573170.181509:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573170.181526:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573170.181530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864029952/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573170.181537:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573170.181540:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573170.181543:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864029952:192.168.202.132@tcp:8:kworker.0 00000008:00100000:3.0:1788573170.181581:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573170.181598:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864030080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573170.181645:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030080:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573170.181664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573170.181673:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573170/real 1788573170] req@ffff92fc6733a300 x1875449864030080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573170.181681:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864030080/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573170.181691:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573170.181694:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573170.181696:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030080:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573170.628883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573170.692834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573171.652938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573171.716840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573172.676889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573172.740831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788573173.700927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573173.764837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573174.724887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573174.788848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573175.300820:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.300829:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573175.300832:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.300841:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4925 00000100:00080000:0.0:1788573175.300851:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.300857:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573175.300892:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864030208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.300916:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.300920:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573175.300925:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.300928:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4925 00000100:00080000:0.0:1788573175.300931:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.300933:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573175.300942:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573175.300951:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b480 x1875449864030336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.300959:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.300962:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573175.300963:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.300966:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4925 00000100:00080000:0.0:1788573175.300967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.300972:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573175.300974:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864030208:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788573175.300976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573175.300986:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b800 x1875449864030464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.300993:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.300995:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573175.300996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.300999:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4925 00000100:00080000:0.0:1788573175.301001:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.301003:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573175.301007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573175.301015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864030592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573175.301021:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573175.301023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573175.301025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573175.301028:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4925 00000100:00080000:0.0:1788573175.301031:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573175.301033:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573175.301036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573175.301041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573175.301047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864030720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573175.301081:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff92fc6704b100 x1875449864030208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573175.301096:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864030208/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573175.301104:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573175.301106:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573175.301109:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864030208:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573175.301114:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030336:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573175.301131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573175.301139:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff92fc6704b480 x1875449864030336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573175.301146:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b480 x1875449864030336/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573175.301151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573175.301153:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573175.301155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030336:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573175.301173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030464:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573175.301183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573175.301189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff92fc6704b800 x1875449864030464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573175.301196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b800 x1875449864030464/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573175.301201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573175.301203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573175.301205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030464:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573175.301209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030592:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573175.301218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573175.301224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff92fc67049500 x1875449864030592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573175.301235:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864030592/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573175.301240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573175.301242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573175.301244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030592:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573175.301247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030720:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573175.301256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573175.301262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573175/real 1788573175] req@ffff92fc67049180 x1875449864030720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573175.301268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864030720/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573175.301273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573175.301275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573175.301277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030720:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573175.748892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573175.812830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573176.772857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573176.836836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573177.796830:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573177.860814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573178.820888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573178.884819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573179.844917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573179.908826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573180.420898:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.420913:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573180.420919:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.420932:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4931 00000100:00080000:0.0:1788573180.420947:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.420956:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573180.421008:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864030848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.421055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.421060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573180.421062:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.421066:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4931 00000100:00080000:0.0:1788573180.421071:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.421076:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573180.421089:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573180.421106:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864030976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.421118:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573180.421124:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864030848:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573180.421125:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573180.421130:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.421137:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4931 00000100:00080000:0.0:1788573180.421141:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.421145:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573180.421151:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573180.421167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864031104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.421178:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.421182:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573180.421184:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573180.421189:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4931 00000100:00080000:0.0:1788573180.421193:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573180.421198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573180.421199:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573180.421208:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573180.421222:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864031232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573180.421239:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573180.421243:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573180.421246:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573180.421249:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff92fc6704bb80 x1875449864030848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573180.421262:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4931 00000100:00100000:1.0:1788573180.421268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864030848/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573180.421269:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573180.421279:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573180.421290:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573180.421294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573180.421299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573180.421304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864030848:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573180.421309:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864031360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573180.421332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030976:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573180.421351:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573180.421364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff92fc67048e00 x1875449864030976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573180.421378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864030976/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573180.421388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573180.421392:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573180.421396:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864030976:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573180.421403:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031104:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573180.421424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573180.421435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff92fc67049c00 x1875449864031104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573180.421443:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864031104/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573180.421449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573180.421451:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573180.421454:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031104:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573180.421458:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031232:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573180.421472:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573180.421478:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff92fc67049180 x1875449864031232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573180.421486:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864031232/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573180.421491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573180.421493:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573180.421494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031232:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573180.421498:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031360:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573180.421508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573180.421514:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573180/real 1788573180] req@ffff92fc67049500 x1875449864031360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573180.421521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864031360/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573180.421526:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573180.421528:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573180.421530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031360:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573180.868848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573180.932829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573181.892887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573181.956791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573182.916846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573182.980865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573183.940891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573184.004792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573184.964853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573185.028850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573185.540873:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.540881:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573185.540885:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.540894:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4936 00000100:00080000:2.0:1788573185.540903:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.540909:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573185.540944:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864031488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.540975:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.540978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573185.540979:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.540982:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4936 00000100:00080000:2.0:1788573185.540984:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.540986:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573185.540994:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573185.541003:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864031616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.541010:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.541012:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573185.541014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.541017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4936 00000100:00080000:2.0:1788573185.541018:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.541021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573185.541025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573185.541026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864031488:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573185.541033:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864031744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.541041:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.541043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573185.541044:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.541047:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4936 00000100:00080000:2.0:1788573185.541050:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.541052:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573185.541055:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573185.541063:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864031872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573185.541069:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573185.541071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573185.541072:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573185.541074:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4936 00000100:00080000:2.0:1788573185.541077:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573185.541079:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573185.541081:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573185.541087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573185.541090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864032000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573185.541128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff92fc4617c000 x1875449864031488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573185.541144:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864031488/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573185.541152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573185.541155:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573185.541158:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864031488:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573185.541163:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031616:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573185.541190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573185.541199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff92fc4617fb80 x1875449864031616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573185.541208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864031616/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573185.541213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573185.541216:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573185.541218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031616:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573185.541237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031744:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573185.541248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573185.541255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff92fc4617c380 x1875449864031744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573185.541262:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864031744/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573185.541272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573185.541274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573185.541276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031744:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573185.541279:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031872:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573185.541290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573185.541296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff92fc4617d180 x1875449864031872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573185.541303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864031872/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573185.541308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573185.541310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573185.541312:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864031872:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573185.541315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032000:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573185.541325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573185.541330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573185/real 1788573185] req@ffff92fc4617e680 x1875449864032000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573185.541338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864032000/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573185.541342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573185.541344:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573185.541347:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032000:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573185.988855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573186.052797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573187.012913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573187.076899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573188.036908:0:112600:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788573188.036941:0:112078:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788573188.036950:0:112078:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788573188.036955:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573188.037036:0:112078:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573188.037043:0:112078:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788573188.037069:0:112078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff92fc67229000 lock: ffff92fc70367c00/0x58a604dcef352d93 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a22179337f expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573188.037081:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573188.037082:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573188.037084:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573188.037165:0:2394:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff92fc6704b800 x1875449864032128/t0(0) o103->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573188.100818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573189.060881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573189.124844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573190.084885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573190.148835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573190.660830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573190.660842:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573190.660846:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573190.660857:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4941 00000100:00080000:2.0:1788573190.660867:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573190.660875:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573190.660910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864032256/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573190.660938:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573190.660950:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573190.660953:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573190.660957:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4941 00000100:00080000:2.0:1788573190.660961:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573190.660966:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573190.660976:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573190.660987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864032256:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573190.660992:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864032384/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573190.661006:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573190.661010:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573190.661012:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573190.661018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4941 00000100:00080000:2.0:1788573190.661022:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573190.661027:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573190.661032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573190.661045:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864032512/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573190.661056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573190.661057:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573190.661063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573190.661065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573190.661071:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4941 00000100:00080000:2.0:1788573190.661076:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573190.661080:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573190.661087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573190.661094:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff92fc4617dc00 x1875449864032256/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573190.661102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864032640/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573190.661115:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864032256/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.661124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573190.661125:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573190.661129:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573190.661135:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864032256:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573190.661135:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573190.661141:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573190.661147:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4941 00000100:00080000:2.0:1788573190.661152:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573190.661157:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573190.661163:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032384:192.168.202.132@tcp:38:kworker.0 00000008:00100000:2.0:1788573190.661163:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573190.661179:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864032768/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573190.661196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573190.661213:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff92fc4617ca80 x1875449864032384/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573190.661245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864032384/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.661252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573190.661254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573190.661257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032384:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573190.661264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573190.661281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573190.661289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff92fc4617c700 x1875449864032512/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573190.661303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864032512/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.661309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573190.661311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573190.661313:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573190.661317:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032640:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573190.661330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573190.661336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff92fc4617f800 x1875449864032640/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573190.661344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864032640/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.661349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573190.661351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573190.661353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032640:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573190.661365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573190.661376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573190.661382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff92fc4617f100 x1875449864032768/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573190.661390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864032768/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.661396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573190.661398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573190.661400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864032768:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573191.108885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573191.172834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573192.132900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573192.196855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573193.156863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573193.220822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573194.180841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573194.244829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573195.204836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573195.268832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573195.780883:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.780899:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573195.780905:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.780918:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4946 00000100:00080000:3.0:1788573195.780931:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.780940:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573195.780986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864032896/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573195.781019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.781022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573195.781024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.781026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4946 00000100:00080000:3.0:1788573195.781029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.781031:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573195.781067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573195.781081:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864033024/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573195.781090:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.781093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573195.781095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.781098:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4946 00000100:00080000:3.0:1788573195.781102:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.781105:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573195.781109:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573195.781123:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864033152/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573195.781125:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864032896:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573195.781135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.781152:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573195.781154:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.781158:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4946 00000100:00080000:3.0:1788573195.781161:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.781163:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573195.781167:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573195.781178:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864033280/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573195.781184:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.781186:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573195.781189:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.781194:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4946 00000100:00080000:3.0:1788573195.781199:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.781205:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573195.781211:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573195.781227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864033408/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573195.781264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033024:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573195.781301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573195.781317:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033280:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573195.781332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033408:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573196.228880:0:112600:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788573196.228920:0:112078:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788573196.228931:0:112078:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 2/0,0 mode: PR/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788573196.228939:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573196.228970:0:112078:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 2/0,0 mode: PR/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573196.229036:0:112078:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 1/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573196.229050:0:112078:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 1/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573196.229057:0:112078:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788573196.229089:0:112078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc51bc2a00/0x58a604dcef352d85 lrc: 0/0,0 mode: --/PR res: [0x24000040b:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793363 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573196.229110:0:112078:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff92fc67229000 lock: ffff92fc70204400/0x58a604dcef352d70 lrc: 0/0,0 mode: --/PR res: [0x200000422:0x1:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xc9a239a221793324 expref: -99 pid: 112108 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788573196.229123:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573196.229126:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788573196.229130:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573196.229140:0:2393:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff92fc6704b480 x1875449864033536/t0(0) o103->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573196.292817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573197.252885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573197.316821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573198.276860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573198.340829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573198.852995:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573198.853041:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573198.853060:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573198.853091:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573198.853106:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573198.853119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff92fc67338380 x1875449864032896/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788573198.853124:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573198.853139:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573198.853155:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864032896/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573198.853169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573198.853176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573198.853182:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864032896:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573198.853197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff92fc6733b100 x1875449864033024/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573198.853210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864033024/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573198.853219:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573198.853222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573198.853225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033024:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573198.853247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff92fc67338a80 x1875449864033152/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573198.853261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864033152/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573198.853269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573198.853273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573198.853276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573198.853283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff92fc6733ad80 x1875449864033280/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573198.853294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864033280/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573198.853301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573198.853304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573198.853307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033280:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573198.853314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573198] req@ffff92fc67339f80 x1875449864033408/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573198.853326:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864033408/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573198.853333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573198.853337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573198.853339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033408:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573199.300899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573199.364786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573200.324825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573200.388837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573200.900872:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573200.900880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573200.900884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573200.900904:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4951 00000100:00080000:1.0:1788573200.900913:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573200.900920:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573200.900955:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864033664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573200.900993:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573200.900997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573200.900999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573200.901004:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4951 00000100:00080000:1.0:1788573200.901014:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573200.901018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573200.901029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573200.901043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864033792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573200.901054:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573200.901058:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573200.901060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573200.901064:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4951 00000100:00080000:1.0:1788573200.901067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573200.901071:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573200.901077:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573200.901088:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864033920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573200.901098:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573200.901102:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573200.901104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573200.901108:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4951 00000100:00080000:1.0:1788573200.901112:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573200.901116:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573200.901123:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573200.901136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864034048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573200.901141:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864033664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573200.901152:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573200.901157:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573200.901159:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573200.901165:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4951 00000100:00080000:1.0:1788573200.901169:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573200.901174:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573200.901179:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573200.901193:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864034176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573200.901219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573200.901262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff92fc67288380 x1875449864033664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573200.901281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864033664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573200.901291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573200.901295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573200.901299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864033664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573200.901307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573200.901329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573200.901342:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff92fc67289c00 x1875449864033792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573200.901355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864033792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573200.901364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573200.901367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573200.901371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573200.901377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573200.901395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573200.901406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff92fc67289500 x1875449864033920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573200.901424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864033920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573200.901433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573200.901436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573200.901440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864033920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573200.901465:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573200.901485:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573200.901495:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff92fc6728bb80 x1875449864034048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573200.901508:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864034048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573200.901516:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573200.901520:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573200.901523:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573200.901530:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573200.901547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573200.901557:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff92fc6728b100 x1875449864034176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573200.901568:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864034176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573200.901582:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573200.901586:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573200.901589:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573201.348901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573201.412795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573202.372859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573202.436836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573203.396898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573203.460812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573204.420864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573204.484821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573205.444927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573205.508788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573206.020830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573206.020842:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573206.020847:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573206.020858:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4956 00000100:00080000:1.0:1788573206.020870:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573206.020879:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573206.020925:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864034304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573206.020965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573206.020970:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573206.020973:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573206.020977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4956 00000100:00080000:1.0:1788573206.020984:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573206.020991:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573206.021005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573206.021028:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864034432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573206.021028:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864034304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573206.021056:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573206.021063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573206.021066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573206.021073:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4956 00000100:00080000:1.0:1788573206.021079:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573206.021085:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573206.021094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573206.021120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864034560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573206.021132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573206.021144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573206.021148:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573206.021152:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573206.021161:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4956 00000100:00080000:1.0:1788573206.021167:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573206.021171:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573206.021178:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573206.021191:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864034688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573206.021202:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573206.021205:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573206.021207:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573206.021210:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4956 00000100:00080000:1.0:1788573206.021213:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573206.021218:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573206.021224:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573206.021235:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864034816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573206.021271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573206/real 1788573206] req@ffff92fc67289180 x1875449864034304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573206.021296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864034304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573206.021310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573206.021314:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573206.021319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864034304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573206.021350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573206.021504:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573206.021524:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573206/real 1788573206] req@ffff92fc67288a80 x1875449864034432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573206.021583:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864034432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573206.021594:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573206.021599:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573206.021605:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573206.021616:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573206.021651:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573206.021667:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573206/real 1788573206] req@ffff92fc67288700 x1875449864034560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573206.021681:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864034560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573206.021690:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573206.021693:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573206.021696:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573206.021703:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573206.021759:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573206.021785:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573206/real 1788573206] req@ffff92fc6728b100 x1875449864034688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573206.021815:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864034688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573206.021824:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573206.021832:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573206.021836:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788573206.021843:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573206.021865:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573206.021876:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573206/real 1788573206] req@ffff92fc6728bb80 x1875449864034816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573206.021888:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864034816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573206.021902:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573206.021906:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573206.021909:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864034816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573206.468916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573206.532893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573207.492931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573207.556847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573208.516883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573208.580843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573209.540922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573209.604835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573210.564892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573210.628850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573211.140868:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573211.140879:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573211.140883:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573211.140892:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4961 00000100:00080000:3.0:1788573211.140901:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573211.140907:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573211.140940:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864034944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573211.140967:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573211.140970:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573211.140971:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573211.140974:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4961 00000100:00080000:3.0:1788573211.140976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573211.140979:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573211.140987:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573211.140996:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864035072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573211.141003:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573211.141006:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573211.141008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573211.141010:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4961 00000100:00080000:3.0:1788573211.141013:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573211.141015:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573211.141018:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573211.141029:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864034944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573211.141029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864035200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573211.141044:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573211.141046:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573211.141046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573211.141051:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4961 00000100:00080000:3.0:1788573211.141054:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573211.141065:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573211.141071:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573211.141086:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864035328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573211.141097:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573211.141101:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573211.141106:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573211.141109:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573211.141113:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4961 00000100:00080000:3.0:1788573211.141118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573211.141122:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573211.141126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573211.141136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864035456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573211.141144:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573211/real 1788573211] req@ffff92fc67339c00 x1875449864034944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573211.141162:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864034944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573211.141185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573211.141190:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573211.141195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864034944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573211.141203:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573211.141230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573211.141243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573211/real 1788573211] req@ffff92fc67339880 x1875449864035072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573211.141253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864035072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573211.141259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573211.141261:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573211.141263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573211.141283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573211.141297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573211.141305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573211/real 1788573211] req@ffff92fc67339180 x1875449864035200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573211.141312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864035200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573211.141317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573211.141319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573211.141321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573211.141324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573211.141336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573211.141342:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573211/real 1788573211] req@ffff92fc67339f80 x1875449864035328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573211.141349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864035328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573211.141354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573211.141356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573211.141358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573211.141361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573211.141371:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573211.141376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573211/real 1788573211] req@ffff92fc6733ad80 x1875449864035456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573211.141384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864035456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573211.141388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573211.141390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573211.141392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573211.588918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573211.652794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573212.612871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573212.676797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573213.637179:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573213.700796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573214.660910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573214.724819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573215.684946:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573215.748816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573216.260777:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.260786:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573216.260790:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.260798:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4966 00000100:00080000:3.0:1788573216.260808:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.260813:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573216.260844:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864035584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.260867:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.260870:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573216.260871:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.260874:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4966 00000100:00080000:3.0:1788573216.260876:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.260879:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573216.260886:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573216.260895:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864035712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.260903:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.260906:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573216.260907:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.260913:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4966 00000100:00100000:0.0:1788573216.260914:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864035584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573216.260916:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.260923:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573216.260926:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573216.260936:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864035840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.260943:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.260946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573216.260947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.260949:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4966 00000100:00080000:3.0:1788573216.260952:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.260955:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573216.260958:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573216.260965:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864035968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573216.260971:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573216.260973:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573216.260975:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573216.260977:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4966 00000100:00080000:3.0:1788573216.260979:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573216.260981:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573216.260985:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573216.260988:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573216.260996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864036096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573216.261027:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff92fc67338a80 x1875449864035584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573216.261042:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864035584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573216.261049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573216.261052:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573216.261055:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864035584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573216.261072:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573216.261087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573216.261095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff92fc6733b100 x1875449864035712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573216.261103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864035712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573216.261109:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573216.261112:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573216.261114:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573216.261117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573216.261127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573216.261133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff92fc67338380 x1875449864035840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573216.261145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864035840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573216.261151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573216.261152:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573216.261154:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573216.261157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573216.261168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573216.261174:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff92fc6733a300 x1875449864035968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573216.261181:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864035968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573216.261186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573216.261188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573216.261190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864035968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573216.261193:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573216.261202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573216.261207:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573216/real 1788573216] req@ffff92fc6733bb80 x1875449864036096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573216.261214:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864036096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573216.261224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573216.261226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573216.261228:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573216.708858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573216.772832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573217.732947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573217.796843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573218.756908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573218.820823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573219.780900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573219.844885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573220.804901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573220.868821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573221.380872:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573221.380885:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573221.380890:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573221.380901:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4971 00000100:00080000:0.0:1788573221.380912:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573221.380921:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573221.380961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864036224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573221.380992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573221.380997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573221.381000:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573221.381005:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4971 00000100:00080000:0.0:1788573221.381010:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573221.381015:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573221.381027:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573221.381027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864036224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573221.381077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864036352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573221.381094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573221.381099:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573221.381102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573221.381107:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4971 00000800:00000100:3.0:1788573221.381127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573221.381136:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573221.381144:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573221.381151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573221.381165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864036480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573221.381172:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff92fc6704b100 x1875449864036224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573221.381201:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573221.381207:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573221.381210:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573221.381213:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864036224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573221.381219:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4971 00000100:00080000:3.0:1788573221.381228:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573221.381230:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573221.381236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788573221.381257:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573221.381259:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864036224:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788573221.381276:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573221.381292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573221.381294:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864036608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573221.381310:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573221.381315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573221.381317:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573221.381321:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573221.381327:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4971 00000100:00100000:3.0:1788573221.381330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff92fc6704ad80 x1875449864036352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573221.381333:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573221.381343:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573221.381348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864036352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573221.381351:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573221.381360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.381363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573221.381367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573221.381372:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864036736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573221.381376:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573221.381398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573221.381407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff92fc6704a680 x1875449864036480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.381417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864036480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.381423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.381425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573221.381427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573221.381435:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573221.381446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573221.381452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff92fc6704aa00 x1875449864036608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.381459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864036608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.381464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.381465:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573221.381467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788573221.381471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573221.381480:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573221.381487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573221/real 1788573221] req@ffff92fc6704a300 x1875449864036736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573221.381494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864036736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573221.381499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573221.381500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573221.381502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573221.828928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573221.892824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573222.852873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573222.916809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573223.876893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573223.940888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573224.900886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573224.964823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573225.924888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573225.988838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573226.500838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.500847:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573226.500851:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.500861:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4977 00000100:00080000:3.0:1788573226.500872:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.500879:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573226.500918:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864036864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.500943:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.500946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573226.500947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.500950:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4977 00000100:00080000:3.0:1788573226.500954:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.500957:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573226.500967:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573226.500980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864036864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573226.500980:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864036992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.500994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.500996:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573226.500997:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.501000:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4977 00000100:00080000:3.0:1788573226.501002:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.501004:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573226.501007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573226.501015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864037120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.501022:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573226.501024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573226.501025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.501027:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4977 00000100:00080000:3.0:1788573226.501030:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.501032:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573226.501036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573226.501044:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864037248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573226.501050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573226.501055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573226.501055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573226.501058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573226.501063:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4977 00000100:00080000:3.0:1788573226.501066:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573226.501068:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573226.501072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573226.501078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864037376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573226.501106:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff92fc67338000 x1875449864036864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573226.501121:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864036864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573226.501130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573226.501133:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573226.501138:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864036864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573226.501171:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573226.501199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573226.501210:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff92fc67339500 x1875449864036992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573226.501219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864036992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573226.501225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573226.501228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573226.501231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864036992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573226.501238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573226.501258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573226.501270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff92fc6733b480 x1875449864037120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573226.501283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864037120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573226.501290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573226.501292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573226.501295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573226.501299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573226.501314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573226.501322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff92fc6733b800 x1875449864037248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573226.501334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864037248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573226.501339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573226.501341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573226.501343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573226.501347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573226.501357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573226.501364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573226/real 1788573226] req@ffff92fc67338700 x1875449864037376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573226.501371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864037376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573226.501376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573226.501378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573226.501379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573226.948856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573227.012792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573227.972958:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573228.036837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573228.996868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573229.060825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573230.020887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573230.084850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573231.044888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573231.108860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573231.620787:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.620794:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573231.620798:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.620807:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4982 00000100:00080000:3.0:1788573231.620815:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.620821:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573231.620852:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864037504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.620875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.620878:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573231.620879:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.620882:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4982 00000100:00080000:3.0:1788573231.620884:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.620886:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573231.620893:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573231.620902:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864037632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.620910:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.620913:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573231.620914:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.620916:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4982 00000100:00080000:3.0:1788573231.620919:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573231.620923:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864037504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573231.620923:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573231.620930:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573231.620937:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864037760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.620944:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.620946:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573231.620947:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.620950:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4982 00000100:00080000:3.0:1788573231.620952:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.620954:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573231.620957:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573231.620966:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864037888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573231.620971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573231.620974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573231.620975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573231.620978:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4982 00000100:00080000:3.0:1788573231.620981:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573231.620983:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788573231.620988:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573231.620989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573231.620998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864038016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573231.621027:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff92fc6733a680 x1875449864037504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.621059:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864037504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.621071:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.621075:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573231.621080:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864037504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573231.621088:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573231.621107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573231.621117:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff92fc67338e00 x1875449864037632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.621126:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864037632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.621132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.621134:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573231.621136:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573231.621160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573231.621180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573231.621192:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff92fc6733aa00 x1875449864037760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.621210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864037760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.621216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.621218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573231.621220:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573231.621224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573231.621239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573231.621246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff92fc67338700 x1875449864037888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.621262:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864037888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.621268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.621270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573231.621272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864037888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573231.621275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573231.621287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573231.621294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff92fc6733b800 x1875449864038016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.621301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864038016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.621328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.621331:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573231.621333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573232.068872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573232.132832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573233.092857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573233.156832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573234.116861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573234.180816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573235.140929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573235.204839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573236.164898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573236.228834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573236.740864:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573236.740872:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573236.740876:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573236.740884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4987 00000100:00080000:1.0:1788573236.740895:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573236.740903:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573236.740943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864038144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573236.740969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573236.740973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573236.740975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573236.740979:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4987 00000100:00080000:1.0:1788573236.740984:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573236.740988:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573236.741000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573236.741014:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864038272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573236.741024:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864038144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573236.741024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573236.741027:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573236.741029:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573236.741032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4987 00000100:00080000:1.0:1788573236.741036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573236.741039:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573236.741044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573236.741053:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864038400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573236.741061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573236.741064:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573236.741065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573236.741068:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4987 00000100:00080000:1.0:1788573236.741079:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573236.741083:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788573236.741087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788573236.741087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573236.741096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864038528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573236.741104:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573236.741107:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573236.741108:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573236.741111:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4987 00000100:00080000:1.0:1788573236.741114:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573236.741117:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573236.741121:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573236.741127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff92fc67289500 x1875449864038144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573236.741129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864038656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573236.741146:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864038144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.741156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.741160:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573236.741164:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864038144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573236.741187:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573236.741208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573236.741230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff92fc67289c00 x1875449864038272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.741244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864038272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.741250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.741252:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573236.741256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573236.741263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573236.741285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573236.741297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff92fc67288380 x1875449864038400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.741310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864038400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.741319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.741322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573236.741325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573236.741329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573236.741345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573236.741352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff92fc6728a680 x1875449864038528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.741360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864038528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.741369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.741371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573236.741374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573236.741377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573236.741387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573236.741393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff92fc6728b800 x1875449864038656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.741400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864038656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.741405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.741406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573236.741408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573237.188922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573237.252828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573238.212892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573238.276820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573239.236885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573239.300860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573240.260918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573240.324800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573241.284806:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573241.348828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573241.860834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573241.860845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573241.860850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573241.860863:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4992 00000100:00080000:1.0:1788573241.860874:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573241.860883:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573241.860922:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864038784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573241.860959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573241.860962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573241.860963:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573241.860966:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4992 00000100:00080000:1.0:1788573241.860970:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573241.860973:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573241.860980:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573241.860992:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864038912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573241.861001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573241.861003:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573241.861005:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573241.861010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864038784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573241.861011:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4992 00000100:00080000:1.0:1788573241.861016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573241.861021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573241.861025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573241.861035:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864039040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573241.861042:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573241.861045:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573241.861046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573241.861048:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4992 00000100:00080000:1.0:1788573241.861051:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573241.861053:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573241.861056:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573241.861062:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864039168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573241.861068:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573241.861070:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573241.861077:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573241.861081:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4992 00000800:00000100:0.0:1788573241.861087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573241.861087:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573241.861094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573241.861098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573241.861106:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864039296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573241.861150:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff92fc67288e00 x1875449864038784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.861173:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864038784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.861185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.861199:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573241.861205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864038784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573241.861230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573241.861265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573241.861282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff92fc67289f80 x1875449864038912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.861296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864038912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.861307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.861311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573241.861316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864038912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573241.861322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573241.861343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573241.861355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff92fc67288000 x1875449864039040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.861370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864039040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.861385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.861389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573241.861392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573241.861399:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573241.861420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573241.861432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff92fc67289880 x1875449864039168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.861449:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864039168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.861459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.861463:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573241.861467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573241.861475:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573241.861496:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573241.861509:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff92fc6728aa00 x1875449864039296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.861523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864039296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.861532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.861535:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573241.861540:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573242.308903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573242.372803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573243.332858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573243.396823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573244.356952:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573244.420787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573245.380868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573245.444855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573246.404908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573246.468849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573246.980849:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.980860:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573246.980866:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.980876:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 4997 00000100:00080000:3.0:1788573246.980887:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.980895:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573246.980941:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864039424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.980971:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.980975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.980978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.980982:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4997 00000100:00080000:3.0:1788573246.980987:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.980991:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573246.981004:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573246.981019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864039552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.981051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.981055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573246.981057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573246.981058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864039424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573246.981062:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4997 00000100:00080000:3.0:1788573246.981068:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.981072:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573246.981076:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573246.981086:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864039680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.981093:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.981097:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.981099:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.981103:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4997 00000100:00080000:3.0:1788573246.981107:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.981112:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573246.981118:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573246.981131:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864039808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.981146:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573246.981147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573246.981150:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573246.981153:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.981157:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 4997 00000100:00080000:3.0:1788573246.981160:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.981163:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573246.981167:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573246.981177:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864039936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573246.981209:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff92fc6733b480 x1875449864039424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.981233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864039424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.981246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.981251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573246.981255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864039424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573246.981284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573246.981309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573246.981323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff92fc67339500 x1875449864039552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.981336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864039552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.981346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.981351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573246.981354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573246.981362:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573246.981383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573246.981395:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff92fc67338000 x1875449864039680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.981408:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864039680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.981416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.981420:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573246.981423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573246.981430:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573246.981453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573246.981464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff92fc6733bb80 x1875449864039808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.981477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864039808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.981487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.981491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573246.981495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573246.981501:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573246.981521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573246.981532:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff92fc6733a300 x1875449864039936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.981544:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864039936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.981552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.981555:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573246.981558:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864039936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573247.428884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573247.492890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573248.452959:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573248.516876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573249.476808:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573249.540834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573250.500881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573250.564849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573251.524812:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573251.588841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573252.100822:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573252.100830:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573252.100834:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573252.100842:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5002 00000100:00080000:3.0:1788573252.100851:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573252.100858:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573252.100891:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864040064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573252.100915:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573252.100918:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573252.100920:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573252.100923:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5002 00000100:00080000:3.0:1788573252.100941:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573252.100946:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573252.100956:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573252.100969:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864040192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573252.100978:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864040064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573252.100986:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573252.100991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573252.100993:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573252.101007:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5002 00000100:00080000:3.0:1788573252.101012:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573252.101017:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573252.101023:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573252.101037:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864040320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573252.101043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573252.101051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573252.101054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573252.101055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573252.101060:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5002 00000100:00080000:3.0:1788573252.101064:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573252.101067:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573252.101071:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573252.101079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864040448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573252.101091:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573252/real 1788573252] req@ffff92fc67338380 x1875449864040064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573252.101094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573252.101101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573252.101104:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573252.101111:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5002 00000100:00080000:3.0:1788573252.101114:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573252.101119:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573252.101122:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573252.101133:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864040576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573252.101135:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864040064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573252.101163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573252.101171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573252.101176:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864040064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573252.101195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573252.101215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573252.101225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573252/real 1788573252] req@ffff92fc6733b100 x1875449864040192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573252.101234:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864040192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573252.101240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573252.101242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573252.101244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573252.101248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573252.101259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573252.101265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573252/real 1788573252] req@ffff92fc67338a80 x1875449864040320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573252.101272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864040320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573252.101278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573252.101279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573252.101281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573252.101285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573252.101295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573252.101301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573252/real 1788573252] req@ffff92fc6733ad80 x1875449864040448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573252.101309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864040448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573252.101314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573252.101315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573252.101317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573252.101321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573252.101331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573252.101336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573252/real 1788573252] req@ffff92fc67339f80 x1875449864040576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573252.101343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864040576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573252.101348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573252.101350:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573252.101352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573252.548899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573252.612799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573253.572969:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573253.636848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573254.596930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573254.660827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573255.620985:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573255.684870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573256.645025:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573256.708877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573257.220901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.220917:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573257.220923:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.220935:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5007 00000100:00080000:3.0:1788573257.220954:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.220963:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573257.221015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864040704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573257.221056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.221061:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573257.221063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.221070:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5007 00000100:00080000:3.0:1788573257.221074:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.221077:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573257.221089:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573257.221102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864040832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573257.221111:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573257.221117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864040704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573257.221117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573257.221119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.221128:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5007 00000100:00080000:3.0:1788573257.221130:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.221133:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573257.221136:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573257.221146:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864040960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573257.221152:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.221155:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573257.221157:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.221159:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5007 00000100:00080000:3.0:1788573257.221162:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.221164:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573257.221168:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573257.221176:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864041088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573257.221183:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573257.221186:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573257.221187:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573257.221190:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5007 00000100:00080000:3.0:1788573257.221192:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573257.221195:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788573257.221201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573257.221201:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573257.221212:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864041216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573257.221260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff92fc67339180 x1875449864040704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573257.221278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864040704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573257.221288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573257.221291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573257.221295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864040704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573257.221318:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573257.221336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573257.221345:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff92fc67339880 x1875449864040832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573257.221354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864040832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573257.221360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573257.221362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573257.221364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573257.221368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573257.221379:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573257.221385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff92fc67339c00 x1875449864040960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573257.221398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864040960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573257.221403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573257.221405:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573257.221406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864040960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573257.221410:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573257.221420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573257.221426:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff92fc67339f80 x1875449864041088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573257.221434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864041088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573257.221439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573257.221441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573257.221443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573257.221446:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573257.221460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573257.221470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573257/real 1788573257] req@ffff92fc6733ad80 x1875449864041216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573257.221483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864041216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573257.221499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573257.221502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573257.221506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573257.668937:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573257.732857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573258.693002:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573258.756885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573259.716938:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573259.780819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573260.740919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573260.804854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573261.764865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573261.828855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573262.340862:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.340871:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573262.340875:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.340886:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5012 00000100:00080000:3.0:1788573262.340897:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.340905:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573262.340944:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864041344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573262.340976:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.340981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573262.340983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.340987:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5012 00000100:00080000:3.0:1788573262.340991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.340996:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573262.341006:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573262.341020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864041472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573262.341026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864041344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573262.341039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.341043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573262.341045:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.341050:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5012 00000100:00080000:3.0:1788573262.341053:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.341058:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573262.341064:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573262.341076:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864041600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573262.341086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.341089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573262.341091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.341095:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5012 00000100:00080000:3.0:1788573262.341099:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.341103:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573262.341109:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573262.341121:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864041728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573262.341127:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041472:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573262.341144:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573262.341147:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573262.341149:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573262.341154:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5012 00000100:00100000:0.0:1788573262.341157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041600:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573262.341161:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573262.341168:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573262.341174:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573262.341180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788573262.341193:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864041856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573262.341240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573262.788997:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573262.852845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573263.812896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573263.876871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573264.836854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573264.900849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788573265.413009:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788573265.413056:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573265.413074:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573265.413119:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573265.413129:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573265.413136:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573265.413143:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788573265.413240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff92fc6733bb80 x1875449864041856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573265.413265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864041856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573265.413278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573265.413284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573265.413288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041856:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573265.413296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff92fc6733a300 x1875449864041728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573265.413308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864041728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573265.413315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573265.413318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573265.413321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573265.413327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff92fc6733b100 x1875449864041472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573265.413339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864041472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573265.413348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573265.413351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573265.413355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573265.413368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff92fc67338380 x1875449864041600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573265.413382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864041600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573265.413391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573265.413395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573265.413398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864041600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573265.413404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573262/real 1788573265] req@ffff92fc67338a80 x1875449864041344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573265.413415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864041344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573265.413423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573265.413426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573265.413429:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864041344:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573265.860876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573265.924839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573266.884897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573266.948857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573267.460892:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.460905:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573267.460911:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.460924:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5018 00000100:00080000:0.0:1788573267.460937:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.460947:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573267.460999:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864041984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573267.461038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.461044:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573267.461045:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.461049:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5018 00000100:00080000:0.0:1788573267.461052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.461056:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573267.461066:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573267.461076:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864042112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573267.461086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.461090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573267.461092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.461097:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5018 00000100:00080000:0.0:1788573267.461100:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.461104:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573267.461110:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573267.461116:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864041984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573267.461127:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864042240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573267.461139:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.461143:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573267.461146:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.461151:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5018 00000100:00080000:0.0:1788573267.461155:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.461159:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573267.461165:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573267.461179:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864042368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573267.461190:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573267.461194:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573267.461197:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573267.461201:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5018 00000100:00080000:0.0:1788573267.461205:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573267.461209:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573267.461215:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573267.461226:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864042496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573267.461244:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573267.461315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff92fc67049880 x1875449864041984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573267.461341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864041984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573267.461354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573267.461358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573267.461363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864041984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573267.461373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573267.461404:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573267.461419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff92fc67048000 x1875449864042112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573267.461433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864042112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573267.461441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573267.461444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573267.461447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573267.461483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573267.461505:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573267.461518:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff92fc67049f80 x1875449864042240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573267.461530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864042240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573267.462226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573267.462233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573267.462238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573267.462247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573267.462286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573267.462305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff92fc67048700 x1875449864042368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573267.462326:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864042368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573267.462336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573267.462339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573267.462343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788573267.462350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573267.462373:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573267.462386:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573267/real 1788573267] req@ffff92fc67048380 x1875449864042496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573267.462398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864042496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573267.462405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573267.462409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573267.462412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573267.908917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573267.972876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573268.932925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573268.996846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573269.956860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573270.020852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573270.980970:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573271.044867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573272.004966:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573272.068897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573272.580890:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573272.580902:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573272.580907:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573272.580917:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5023 00000100:00080000:1.0:1788573272.580930:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573272.580938:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573272.580997:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864042624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573272.581035:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573272.581040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573272.581043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573272.581049:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5023 00000100:00080000:1.0:1788573272.581054:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573272.581058:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573272.581071:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573272.581086:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864042752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573272.581087:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864042624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573272.581098:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573272.581103:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573272.581106:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573272.581111:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5023 00000100:00080000:1.0:1788573272.581115:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573272.581120:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573272.581126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573272.581136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864042880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573272.581144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573272.581148:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573272.581151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573272.581155:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5023 00000100:00080000:1.0:1788573272.581159:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573272.581163:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573272.581167:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573272.581171:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573272.581180:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864043008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573272.581188:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573272.581191:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573272.581193:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573272.581197:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5023 00000100:00080000:1.0:1788573272.581199:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573272.581202:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573272.581206:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573272.581217:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864043136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573272.581225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff92fc6728a300 x1875449864042624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.581245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864042624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.581258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.581262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573272.581267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864042624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573272.581294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573272.581314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573272.581324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff92fc6728ad80 x1875449864042752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.581333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864042752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.581339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.581341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573272.581343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573272.581347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573272.581362:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573272.581373:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff92fc6728b480 x1875449864042880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.581385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864042880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.581394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.581398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573272.581402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864042880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573272.581409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573272.581429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573272.581437:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff92fc6728aa00 x1875449864043008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.581445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864043008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.581451:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.581453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573272.581455:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573272.581459:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573272.581470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573272.581476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff92fc67289880 x1875449864043136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.581483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864043136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.581488:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.581489:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573272.581491:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573273.028888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573273.092854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573274.052818:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573274.116825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573275.076979:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573275.140774:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573276.100888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573276.164825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573277.124947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573277.188835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573277.700869:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.700882:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573277.700887:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.700898:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5028 00000100:00080000:2.0:1788573277.700910:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.700918:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573277.700960:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864043264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.700985:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.700988:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573277.700990:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.700994:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5028 00000100:00080000:2.0:1788573277.700998:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.701003:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573277.701014:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573277.701025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864043264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573277.701035:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864043392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.701048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.701052:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573277.701054:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.701059:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5028 00000100:00080000:2.0:1788573277.701062:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.701065:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573277.701069:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573277.701078:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864043520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.701084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.701091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573277.701092:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.701096:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5028 00000100:00080000:2.0:1788573277.701099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573277.701106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573277.701110:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573277.701115:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573277.701124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864043648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.701136:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.701138:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573277.701140:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.701143:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5028 00000100:00080000:2.0:1788573277.701145:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.701151:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573277.701155:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573277.701165:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864043776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573277.701177:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff92fc4617ed80 x1875449864043264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573277.701202:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864043264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573277.701212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573277.701217:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573277.701221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864043264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573277.701246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573277.701271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573277.701283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff92fc4617f480 x1875449864043392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573277.701304:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864043392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573277.701313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573277.701316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573277.701319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573277.701325:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573277.701350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573277.701369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff92fc4617f100 x1875449864043520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573277.701382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864043520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573277.701392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573277.701396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573277.701400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573277.701408:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573277.701432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573277.701465:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff92fc4617f800 x1875449864043648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573277.701474:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864043648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573277.701479:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573277.701481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573277.701483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573277.701487:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573277.701500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573277.701507:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573277] req@ffff92fc4617c700 x1875449864043776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573277.701518:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864043776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573277.701522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573277.701523:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573277.701525:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864043776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573278.148891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573278.212795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573279.172901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573279.236791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573280.196848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573280.260794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573281.220879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573281.284803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573282.244872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573282.308778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573282.820898:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573282.820907:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573282.820910:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573282.820919:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5033 00000100:00080000:3.0:1788573282.820926:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573282.820938:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573282.820966:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864043904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573282.820990:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573282.820993:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573282.820994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573282.820997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5033 00000100:00080000:3.0:1788573282.820999:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573282.821001:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573282.821009:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573282.821018:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864044032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573282.821024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573282.821027:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573282.821028:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573282.821030:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5033 00000100:00080000:3.0:1788573282.821032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573282.821035:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573282.821037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573282.821043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864044160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573282.821053:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864043904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573282.821054:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573282.821073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573282.821075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573282.821078:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5033 00000100:00080000:3.0:1788573282.821085:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573282.821088:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573282.821091:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573282.821101:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864044288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573282.821107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573282.821110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573282.821112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573282.821115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5033 00000100:00080000:3.0:1788573282.821117:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573282.821119:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573282.821122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573282.821127:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864044416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573282.821141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573282.821187:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff92fc67338000 x1875449864043904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573282.821202:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864043904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573282.821211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573282.821214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573282.821218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864043904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573282.821223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573282.821238:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573282.821245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff92fc67339500 x1875449864044032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573282.821252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864044032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573282.821262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573282.821265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573282.821267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573282.821294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573282.821305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573282.821311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff92fc6733b480 x1875449864044160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573282.821318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864044160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573282.821323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573282.821325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573282.821327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573282.821330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573282.821341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573282.821346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff92fc6733b800 x1875449864044288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573282.821352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864044288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573282.821356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573282.821361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573282.821364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573282.821368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573282.821378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573282.821384:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff92fc67338700 x1875449864044416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573282.821391:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864044416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573282.821395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573282.821397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573282.821398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573283.268908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573283.332834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573284.292915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573284.356881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573285.316889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573285.380866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573286.340822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573286.404851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573287.364892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573287.428867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573287.940799:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.940807:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573287.940811:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.940821:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5038 00000100:00080000:0.0:1788573287.940830:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.940836:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573287.940869:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864044544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573287.940893:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.940896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573287.940898:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.940903:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5038 00000100:00080000:0.0:1788573287.940906:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.940911:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573287.940920:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573287.940926:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864044544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573287.940936:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864044672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573287.940947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.940950:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573287.940952:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.940956:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5038 00000100:00080000:0.0:1788573287.940977:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.940981:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573287.940986:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573287.940990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573287.941000:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864044800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573287.941011:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.941014:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573287.941016:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.941020:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5038 00000100:00080000:0.0:1788573287.941024:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.941028:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573287.941033:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573287.941034:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff92fc67048a80 x1875449864044544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573287.941047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864044928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573287.941052:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864044544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573287.941071:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573287.941072:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573287.941078:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573287.941079:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788573287.941083:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573287.941085:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864044544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573287.941090:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5038 00000100:00080000:0.0:1788573287.941094:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.941098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573287.941102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573287.941106:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573287.941115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864045056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573287.941126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573287.941137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff92fc67048380 x1875449864044672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.941148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864044672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.941156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.941159:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573287.941162:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573287.941168:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573287.941186:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573287.941195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff92fc67048700 x1875449864044800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.941203:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864044800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.941208:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.941210:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573287.941212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573287.941215:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573287.941227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573287.941232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff92fc67049f80 x1875449864044928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.941238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864044928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.941243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.941245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573287.941247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864044928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573287.941264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573287.941274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573287.941280:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff92fc67048000 x1875449864045056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573287.941286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864045056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573287.941291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573287.941293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573287.941294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573288.388828:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573288.452833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573289.412926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573289.476824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573290.436872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573290.500825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573291.460901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573291.524821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573292.484925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573292.548857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573293.060836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573293.060848:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573293.060861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573293.060872:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5043 00000100:00080000:3.0:1788573293.060885:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573293.060893:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573293.060940:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864045184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573293.060967:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573293.060970:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573293.060972:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573293.060975:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5043 00000100:00080000:3.0:1788573293.060977:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573293.060979:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573293.060990:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573293.061000:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864045312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573293.061009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573293.061012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573293.061013:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573293.061016:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5043 00000100:00080000:3.0:1788573293.061021:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573293.061022:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864045184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573293.061027:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573293.061032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573293.061042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864045440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573293.061048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573293.061051:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573293.061052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573293.061055:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5043 00000100:00080000:3.0:1788573293.061057:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573293.061060:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573293.061064:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573293.061073:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864045568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573293.061079:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573293.061081:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573293.061083:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573293.061086:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5043 00000100:00080000:3.0:1788573293.061088:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573293.061091:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573293.061094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573293.061099:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573293.061105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864045696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573293.061149:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573293/real 1788573293] req@ffff92fc6733aa00 x1875449864045184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573293.061166:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864045184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573293.061176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573293.061179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573293.061182:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864045184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573293.061216:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573293.061233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573293.061242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573293/real 1788573293] req@ffff92fc67338e00 x1875449864045312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573293.061251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864045312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573293.061257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573293.061260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573293.061264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573293.061271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573293.061302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573293.061315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573293/real 1788573293] req@ffff92fc6733a680 x1875449864045440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573293.061327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864045440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573293.061334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573293.061336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573293.061338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573293.061343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573293.061380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573293.061389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573293/real 1788573293] req@ffff92fc67338700 x1875449864045568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573293.061398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864045568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573293.061404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573293.061406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573293.061408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573293.061412:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573293.061432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573293.061439:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573293/real 1788573293] req@ffff92fc6733b800 x1875449864045696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573293.061447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864045696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573293.061453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573293.061455:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573293.061457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573293.508952:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573293.572864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573294.532910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573294.596893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573295.556931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573295.620830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573296.580860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573296.644841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573297.604973:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573297.668800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573298.180938:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573298.180954:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573298.180959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573298.180971:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5048 00000100:00080000:0.0:1788573298.180989:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573298.181002:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573298.181039:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864045824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573298.181065:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573298.181068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573298.181070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573298.181073:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5048 00000100:00080000:0.0:1788573298.181076:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573298.181079:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573298.181087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573298.181098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864045952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573298.181106:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573298.181112:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573298.181112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864045824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573298.181116:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573298.181124:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5048 00000100:00080000:0.0:1788573298.181126:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573298.181128:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573298.181131:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573298.181141:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864046080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573298.181152:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573298.181155:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573298.181156:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573298.181158:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5048 00000100:00080000:0.0:1788573298.181175:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573298.181177:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573298.181180:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573298.181188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573298.181191:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864046208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573298.181198:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573298.181201:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573298.181202:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573298.181205:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5048 00000100:00080000:0.0:1788573298.181209:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573298.181212:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573298.181214:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573298.181221:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864046336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573298.181237:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff92fc67049880 x1875449864045824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573298.181261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864045824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573298.181281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573298.181285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573298.181288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864045824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573298.181329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573298.181356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573298.181367:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff92fc6704a300 x1875449864045952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573298.181376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864045952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573298.181383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573298.181386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573298.181390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864045952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573298.181397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573298.181416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573298.181429:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff92fc6704aa00 x1875449864046080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573298.181442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864046080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573298.181448:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573298.181450:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573298.181452:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573298.181457:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573298.181472:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573298.181479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff92fc6704a680 x1875449864046208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573298.181491:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864046208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573298.181499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573298.181502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573298.181506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573298.181512:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573298.181532:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573298.181542:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573298/real 1788573298] req@ffff92fc6704ad80 x1875449864046336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573298.181550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864046336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573298.181557:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573298.181560:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573298.181563:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573298.628905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573298.692800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573299.652914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573299.716879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573300.676896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573300.740890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573301.700937:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573301.764878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573302.724878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573302.788898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573303.300868:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573303.300884:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573303.300890:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573303.300902:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5053 00000100:00080000:3.0:1788573303.300922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573303.300931:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573303.300981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864046464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573303.301007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573303.301010:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573303.301011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573303.301014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5053 00000100:00080000:3.0:1788573303.301017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573303.301019:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573303.301032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573303.301041:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864046592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573303.301048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573303.301051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573303.301053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573303.301056:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5053 00000100:00080000:3.0:1788573303.301058:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573303.301060:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573303.301065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573303.301072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864046720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573303.301081:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864046464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573303.301082:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573303.301087:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573303.301090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573303.301096:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5053 00000100:00080000:3.0:1788573303.301098:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573303.301100:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573303.301103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573303.301110:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864046848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573303.301116:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573303.301118:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573303.301119:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573303.301122:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5053 00000100:00080000:3.0:1788573303.301125:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573303.301127:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573303.301131:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573303.301137:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864046976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573303.301168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573303.301212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff92fc6733b480 x1875449864046464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573303.301230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864046464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573303.301267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573303.301271:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573303.301274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864046464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573303.301280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573303.301299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573303.301308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff92fc67339500 x1875449864046592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573303.301317:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864046592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573303.301323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573303.301325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573303.301327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573303.301350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573303.301361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573303.301368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff92fc67338000 x1875449864046720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573303.301375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864046720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573303.301386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573303.301388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573303.301390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573303.301393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573303.301405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573303.301411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff92fc67338a80 x1875449864046848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573303.301418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864046848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573303.301423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573303.301425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573303.301427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573303.301430:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573303.301441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573303.301447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573303/real 1788573303] req@ffff92fc67338380 x1875449864046976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573303.301454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864046976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573303.301459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573303.301460:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573303.301463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864046976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573303.748885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573303.812835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573304.772893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573304.836836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573305.796894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573305.860850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573306.820910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573306.884879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573307.844854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573307.908830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573308.420847:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573308.420855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573308.420859:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573308.420868:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5059 00000100:00080000:0.0:1788573308.420877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573308.420883:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573308.420916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864047104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573308.420945:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573308.420950:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573308.420952:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573308.420957:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5059 00000100:00080000:0.0:1788573308.420961:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573308.420965:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573308.420976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573308.420991:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864047232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573308.420991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864047104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573308.421011:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573308.421015:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573308.421017:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573308.421021:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5059 00000100:00080000:0.0:1788573308.421025:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573308.421030:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573308.421035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573308.421049:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864047360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573308.421059:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573308.421062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573308.421066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573308.421068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573308.421072:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5059 00000100:00080000:0.0:1788573308.421078:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573308.421081:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573308.421085:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573308.421096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864047488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573308.421100:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff92fc6704b100 x1875449864047104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573308.421109:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573308.421114:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573308.421115:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573308.421118:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5059 00000100:00080000:0.0:1788573308.421121:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573308.421124:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573308.421128:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573308.421132:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864047104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573308.421136:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864047616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573308.421149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573308.421154:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573308.421158:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864047104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573308.421190:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573308.421208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573308.421217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff92fc67049500 x1875449864047232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573308.421226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864047232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573308.421232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573308.421234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573308.421236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573308.421240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573308.421250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573308.421257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff92fc67049180 x1875449864047360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573308.421264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864047360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573308.421270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573308.421272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573308.421274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573308.421277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573308.421287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573308.421293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff92fc67049c00 x1875449864047488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573308.421300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864047488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573308.421311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573308.421313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573308.421315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573308.421319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573308.421330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573308.421337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573308/real 1788573308] req@ffff92fc67048e00 x1875449864047616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573308.421344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864047616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573308.421349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573308.421351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573308.421353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573308.868908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573308.932836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573309.892895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573309.956816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573310.916886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573310.980890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573311.940895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573312.004828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573312.964919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573313.028921:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573313.540847:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573313.540857:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573313.540862:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573313.540874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5064 00000100:00080000:3.0:1788573313.540886:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573313.540895:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573313.540930:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864047744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573313.540960:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573313.540965:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573313.540968:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573313.540974:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5064 00000100:00080000:3.0:1788573313.540989:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573313.541002:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573313.541010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864047744:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573313.541014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573313.541041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864047872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573313.541057:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573313.541061:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573313.541066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573313.541073:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573313.541074:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5064 00000100:00080000:3.0:1788573313.541084:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573313.541097:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573313.541104:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573313.541145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864048000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573313.541155:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff92fc6733b100 x1875449864047744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573313.541155:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573313.541165:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573313.541168:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573313.541183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864047744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573313.541193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573313.541197:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573313.541201:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864047744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573313.541226:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047872:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573313.541243:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5064 00000800:00000100:0.0:1788573313.541248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573313.541248:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573313.541255:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573313.541262:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573313.541263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff92fc6733a300 x1875449864047872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573313.541276:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864048128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573313.541277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864047872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573313.541289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573313.541292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573313.541294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864047872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573313.541300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573313.541321:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573313.541334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff92fc6733bb80 x1875449864048000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573313.541348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864048000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573313.541357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573313.541359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573313.541361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048000:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573313.541414:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573313.541417:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573313.541419:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573313.541423:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5064 00000100:00100000:0.0:1788573313.541453:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048128:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788573313.541460:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573313.541466:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573313.541471:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573313.541484:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864048256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573313.541494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573313.541540:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff92fc6733ad80 x1875449864048128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573313.541550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864048128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573313.541556:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573313.541558:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573313.541560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573313.541569:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573313.541584:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573313.541591:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573313/real 1788573313] req@ffff92fc67339f80 x1875449864048256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573313.541599:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864048256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573313.541604:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573313.541606:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573313.541608:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788573313.988939:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573314.052842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573315.012862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573315.076874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573316.036899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573316.100847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573317.060898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573317.124870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573318.084961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573318.148829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573318.660874:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.660888:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573318.660894:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.660907:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5069 00000100:00080000:1.0:1788573318.660920:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.660929:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573318.660979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864048384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.661027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.661032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573318.661034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.661038:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5069 00000100:00080000:1.0:1788573318.661042:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.661048:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573318.661076:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573318.661093:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864048512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.661108:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573318.661111:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864048384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573318.661113:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573318.661123:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.661130:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5069 00000100:00080000:1.0:1788573318.661136:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.661140:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573318.661147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573318.661164:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864048640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.661175:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.661179:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573318.661182:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.661187:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5069 00000100:00080000:1.0:1788573318.661192:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573318.661199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573318.661205:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573318.661213:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573318.661230:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864048768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.661243:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573318.661245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff92fc67288000 x1875449864048384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573318.661252:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573318.661261:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.661276:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5069 00000100:00080000:1.0:1788573318.661284:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573318.661286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864048384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573318.661299:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788573318.661310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788573318.661311:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573318.661317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573318.661334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864048384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573318.661336:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864048896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573318.661363:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573318.661383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573318.661394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff92fc67289f80 x1875449864048512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.661403:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864048512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.661409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.661412:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573318.661414:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573318.661418:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573318.661429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573318.661435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff92fc67288e00 x1875449864048640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.661443:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864048640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.661459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.661462:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573318.661464:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573318.661468:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573318.661480:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573318.661487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff92fc6728b800 x1875449864048768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.661494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864048768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.661504:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.661507:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573318.661509:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573318.661513:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573318.661524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573318.661531:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff92fc6728a680 x1875449864048896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.661538:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864048896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.661544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.661547:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573318.661551:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864048896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573319.108854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573319.172835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573320.132911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573320.196841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573321.156897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573321.220844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573322.180947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573322.244896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573323.204879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573323.268873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573323.780845:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573323.780857:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573323.780862:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573323.780873:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5074 00000100:00080000:3.0:1788573323.780884:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573323.780892:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573323.780948:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864049024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573323.780979:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573323.780982:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573323.780984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573323.780997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5074 00000100:00080000:3.0:1788573323.781001:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573323.781004:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573323.781016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573323.781029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864049152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573323.781038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864049024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573323.781043:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573323.781049:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573323.781051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573323.781056:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5074 00000100:00080000:3.0:1788573323.781061:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573323.781066:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573323.781073:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573323.781089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864049280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573323.781097:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573323.781099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573323.781101:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573323.781104:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5074 00000100:00080000:3.0:1788573323.781108:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573323.781110:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573323.781115:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573323.781128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573323.781132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864049408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573323.781144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573323.781149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573323.781151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573323.781156:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5074 00000100:00080000:3.0:1788573323.781161:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573323.781166:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573323.781173:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573323.781183:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff92fc67339c00 x1875449864049024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573323.781187:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864049536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573323.781218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864049024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.781229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.781232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573323.781236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864049024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573323.781259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573323.781279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573323.781289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff92fc67339880 x1875449864049152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.781297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864049152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.781305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.781307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573323.781310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573323.781313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573323.781324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573323.781330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff92fc67339180 x1875449864049280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.781338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864049280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.781343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.781345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573323.781347:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573323.781351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573323.781375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573323.781382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff92fc67339f80 x1875449864049408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.781389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864049408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.781395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.781397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573323.781400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573323.781404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573323.781416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573323.781422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff92fc6733ad80 x1875449864049536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.781429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864049536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.781434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.781436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573323.781439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573324.228915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573324.292875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573325.252845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573325.316821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573326.276827:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573326.340830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573327.300847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573327.364846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573328.324919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573328.388853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573328.900909:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573328.900923:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573328.900929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573328.900973:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5079 00000100:00080000:1.0:1788573328.900986:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573328.900996:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573328.901047:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864049664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573328.901090:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573328.901094:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573328.901097:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573328.901101:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5079 00000100:00080000:1.0:1788573328.901104:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573328.901108:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573328.901121:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573328.901134:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864049792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573328.901145:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573328.901149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573328.901151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573328.901156:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5079 00000100:00080000:1.0:1788573328.901160:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573328.901164:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573328.901170:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573328.901179:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864049664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573328.901187:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864049920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573328.901202:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573328.901205:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573328.901208:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573328.901213:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5079 00000100:00080000:1.0:1788573328.901216:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573328.901220:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573328.901226:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573328.901240:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864050048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573328.901251:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573328.901255:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573328.901257:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573328.901262:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5079 00000100:00080000:1.0:1788573328.901266:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573328.901271:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573328.901275:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573328.901295:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864050176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573328.901307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573328.901355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573328.901386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573328.901411:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573329.348865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573329.412847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573330.372869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573330.436860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573331.396858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573331.460811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573331.973115:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573331.973166:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573331.973184:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573331.973226:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573331.973236:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573331.973246:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573331.973255:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573331.973282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff92fc67289500 x1875449864049920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573331.973311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864049920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573331.973325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573331.973330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573331.973335:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573331.973344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff92fc6728bb80 x1875449864050048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573331.973357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864050048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573331.973366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573331.973370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573331.973374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573331.973381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff92fc6728b100 x1875449864050176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573331.973400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864050176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573331.973408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573331.973412:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573331.973415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050176:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573331.973421:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff92fc67288380 x1875449864049664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573331.973434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864049664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573331.973442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573331.973446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573331.973449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864049664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573331.973456:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573331] req@ffff92fc67289c00 x1875449864049792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573331.973468:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864049792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573331.973478:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573331.973482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573331.973485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864049792:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788573332.420886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573332.484889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573333.444858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573333.508785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573334.020882:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573334.020891:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573334.020894:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573334.020904:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5084 00000100:00080000:3.0:1788573334.020913:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573334.020919:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573334.020973:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864050304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573334.021008:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573334.021011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573334.021013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573334.021018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5084 00000100:00080000:3.0:1788573334.021022:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573334.021027:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573334.021037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573334.021052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864050304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573334.021055:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864050432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573334.021070:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573334.021074:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573334.021082:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573334.021089:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5084 00000100:00080000:3.0:1788573334.021093:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573334.021097:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573334.021103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573334.021116:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864050560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573334.021131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573334.021140:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573334.021144:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573334.021146:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573334.021152:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5084 00000100:00080000:3.0:1788573334.021156:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573334.021160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573334.021165:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573334.021178:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864050688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573334.021187:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573334.021191:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573334.021194:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573334.021196:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573334/real 1788573334] req@ffff92fc6733bb80 x1875449864050304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573334.021201:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5084 00000100:00080000:3.0:1788573334.021207:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573334.021214:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573334.021221:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573334.021226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864050304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573334.021237:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864050816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573334.021241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573334.021251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573334.021260:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864050304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573334.021288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573334.021314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573334.021335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573334/real 1788573334] req@ffff92fc6733a300 x1875449864050432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573334.021348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864050432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573334.021357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573334.021362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573334.021365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573334.021371:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573334.021393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573334.021405:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573334/real 1788573334] req@ffff92fc6733b100 x1875449864050560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573334.021419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864050560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573334.021428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573334.021432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573334.021435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573334.021440:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573334.021469:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573334.021482:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573334/real 1788573334] req@ffff92fc67338380 x1875449864050688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573334.021493:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864050688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573334.021500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573334.021503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573334.021506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573334.021513:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573334.021531:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573334.021541:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573334/real 1788573334] req@ffff92fc67338a80 x1875449864050816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573334.021551:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864050816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573334.021559:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573334.021562:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573334.021565:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864050816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573334.468855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573334.532844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573335.492917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573335.556821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573336.516904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573336.580870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573337.540867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573337.604777:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573338.564822:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573338.628851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573339.140857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573339.140878:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573339.140883:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573339.140894:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5089 00000100:00080000:3.0:1788573339.140907:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573339.140915:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573339.140977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864050944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573339.141015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573339.141020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573339.141022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573339.141027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5089 00000100:00080000:3.0:1788573339.141031:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573339.141036:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573339.141049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573339.141080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864051072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573339.141095:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573339.141099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573339.141103:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864050944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573339.141104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573339.141113:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5089 00000100:00080000:3.0:1788573339.141117:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573339.141120:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573339.141126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573339.141138:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864051200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573339.141146:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573339.141149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573339.141151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573339.141154:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5089 00000100:00080000:3.0:1788573339.141157:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573339.141159:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573339.141162:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573339.141171:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864051328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573339.141180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573339.141180:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573339.141185:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573339.141187:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573339.141191:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5089 00000100:00080000:3.0:1788573339.141193:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573339.141196:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573339.141199:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573339.141207:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864051456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573339.141234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573339/real 1788573339] req@ffff92fc67338000 x1875449864050944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573339.141259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864050944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573339.141280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573339.141284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573339.141288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864050944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573339.141312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573339.141331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573339.141341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573339/real 1788573339] req@ffff92fc67339500 x1875449864051072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573339.141350:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864051072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573339.141357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573339.141359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573339.141361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573339.141365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573339.141376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573339.141382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573339/real 1788573339] req@ffff92fc6733b480 x1875449864051200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573339.141389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864051200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573339.141395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573339.141397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573339.141399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573339.141403:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573339.141414:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573339.141420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573339/real 1788573339] req@ffff92fc6733b800 x1875449864051328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573339.141429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864051328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573339.141438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573339.141442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573339.141445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573339.141451:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573339.141485:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573339.141494:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573339/real 1788573339] req@ffff92fc67338700 x1875449864051456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573339.141502:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864051456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573339.141508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573339.141511:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573339.141515:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573339.588850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573339.652827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573340.612830:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573340.676824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573341.636870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573341.700838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573342.660830:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573342.724821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573343.684874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573343.748795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573344.260884:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.260896:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573344.260901:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.260913:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5094 00000100:00080000:1.0:1788573344.260925:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.260947:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573344.260995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864051584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.261024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.261027:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573344.261029:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.261032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5094 00000100:00080000:1.0:1788573344.261035:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.261037:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573344.261048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573344.261057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864051712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.261066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.261068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573344.261070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573344.261076:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864051584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573344.261076:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5094 00000100:00080000:1.0:1788573344.261081:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.261087:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573344.261091:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573344.261098:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864051840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.261109:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.261112:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573344.261113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.261116:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5094 00000100:00080000:1.0:1788573344.261117:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.261120:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573344.261123:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573344.261131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864051968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573344.261136:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573344.261138:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573344.261139:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573344.261142:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5094 00000100:00080000:1.0:1788573344.261154:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573344.261160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788573344.261166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788573344.261170:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573344.261187:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864052096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573344.261222:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff92fc67289c00 x1875449864051584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573344.261245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864051584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573344.261259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573344.261264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573344.261268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864051584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573344.261299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573344.261324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573344.261338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff92fc67288380 x1875449864051712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573344.261351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864051712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573344.261361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573344.261365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573344.261371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573344.261377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573344.261396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573344.261406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff92fc6728b100 x1875449864051840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573344.261418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864051840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573344.261438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573344.261441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573344.261445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573344.261451:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573344.261471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573344.261482:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff92fc6728bb80 x1875449864051968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573344.261494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864051968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573344.261502:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573344.261505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573344.261508:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864051968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573344.261515:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573344.261533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573344.261544:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573344/real 1788573344] req@ffff92fc67289500 x1875449864052096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573344.261557:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864052096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573344.261565:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573344.261568:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573344.261572:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573344.708812:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573344.772798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573345.732902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573345.796824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573346.756990:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573346.820837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573347.780831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573347.844780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573348.804873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573348.868790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573349.380880:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573349.380887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573349.380891:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573349.380901:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5099 00000100:00080000:3.0:1788573349.380910:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573349.380917:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573349.380965:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864052224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573349.380989:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573349.380992:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573349.380994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573349.380997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5099 00000100:00080000:3.0:1788573349.381001:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573349.381005:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573349.381026:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573349.381041:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864052224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573349.381048:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864052352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573349.381064:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573349.381068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573349.381070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573349.381074:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5099 00000100:00080000:3.0:1788573349.381078:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573349.381082:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573349.381087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573349.381096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864052480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573349.381103:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573349.381105:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573349.381107:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573349.381110:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5099 00000800:00000100:1.0:1788573349.381116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573349.381116:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573349.381122:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573349.381130:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573349.381145:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864052608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573349.381152:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573349.381154:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573349.381156:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573349.381160:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5099 00000100:00080000:3.0:1788573349.381163:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573349.381166:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573349.381170:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573349.381176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff92fc6733a680 x1875449864052224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573349.381183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864052736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573349.381225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864052224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573349.381234:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573349.381237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573349.381239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864052224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573349.381259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573349.381276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573349.381286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff92fc67338e00 x1875449864052352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573349.381297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864052352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573349.381303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573349.381306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573349.381308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573349.381313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573349.381326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573349.381332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff92fc6733aa00 x1875449864052480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573349.381342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864052480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573349.381357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573349.381361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573349.381364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573349.381370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573349.381391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573349.381413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff92fc67338700 x1875449864052608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573349.381424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864052608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573349.381430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573349.381432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573349.381434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573349.381438:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573349.381453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573349.381463:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573349/real 1788573349] req@ffff92fc6733b800 x1875449864052736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573349.381470:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864052736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573349.381475:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573349.381477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573349.381479:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573349.828860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573349.892848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573350.852854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573350.916833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573351.876850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573351.940827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573352.900867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573352.964797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573353.924847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573353.988836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573354.500817:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.500825:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573354.500829:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.500838:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5105 00000100:00080000:3.0:1788573354.500847:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.500864:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573354.500896:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864052864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573354.500919:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.500924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573354.500925:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.500928:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5105 00000100:00080000:3.0:1788573354.500955:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.500959:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573354.500967:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573354.500978:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864052992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573354.500988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864052864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573354.500989:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.500996:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573354.501000:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.501004:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5105 00000100:00080000:3.0:1788573354.501006:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.501008:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573354.501012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573354.501020:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864053120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573354.501026:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.501029:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573354.501030:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.501032:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5105 00000100:00080000:3.0:1788573354.501034:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.501037:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573354.501041:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573354.501049:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864053248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573354.501055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573354.501057:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573354.501058:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573354.501061:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5105 00000100:00080000:3.0:1788573354.501064:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573354.501066:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573354.501068:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573354.501072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573354.501079:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864053376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573354.501111:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff92fc6733b480 x1875449864052864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.501127:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864052864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.501135:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.501138:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573354.501141:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864052864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573354.501160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573354.501175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573354.501184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff92fc67339500 x1875449864052992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.501192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864052992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.501198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.501200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573354.501203:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864052992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573354.501206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573354.501216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573354.501223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff92fc67338000 x1875449864053120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.501239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864053120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.501245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.501247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573354.501249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573354.501252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573354.501264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573354.501271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff92fc67338a80 x1875449864053248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.501279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864053248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.501291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.501293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573354.501295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573354.501299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573354.501310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573354.501316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573354/real 1788573354] req@ffff92fc67338380 x1875449864053376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573354.501324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864053376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573354.501329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573354.501331:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573354.501333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573354.948865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573355.012784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573355.972894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573356.036901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573356.997003:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573357.060823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573358.020831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573358.084798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573359.044912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573359.108836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573359.620850:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573359.620859:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573359.620862:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573359.620871:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5110 00000100:00080000:3.0:1788573359.620881:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573359.620888:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573359.620921:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864053504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573359.620955:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573359.620958:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573359.620960:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573359.620967:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5110 00000100:00080000:3.0:1788573359.620971:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573359.620975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573359.620986:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573359.621003:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864053632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573359.621012:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864053504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573359.621020:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573359.621025:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573359.621027:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573359.621032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5110 00000100:00080000:3.0:1788573359.621036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573359.621041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573359.621046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573359.621060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864053760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573359.621070:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573359.621073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573359.621075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573359.621081:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5110 00000800:00000100:1.0:1788573359.621085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573359.621088:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573359.621093:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573359.621098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573359.621115:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864053888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573359.621130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573359.621133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff92fc6733b100 x1875449864053504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573359.621135:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573359.621139:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573359.621148:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5110 00000100:00080000:3.0:1788573359.621155:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573359.621160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573359.621166:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864053504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573359.621167:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573359.621181:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864054016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573359.621183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573359.621194:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573359.621200:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864053504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573359.621227:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573359.621252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573359.621266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff92fc6733a300 x1875449864053632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573359.621280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864053632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573359.621289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573359.621293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573359.621297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573359.621304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573359.621323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573359.621334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff92fc6733bb80 x1875449864053760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573359.621347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864053760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573359.621356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573359.621359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573359.621362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573359.621369:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573359.621390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573359.621416:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff92fc6733ad80 x1875449864053888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573359.621430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864053888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573359.621438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573359.621441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573359.621444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864053888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573359.621451:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573359.621471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573359.621483:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff92fc67339f80 x1875449864054016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573359.621495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864054016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573359.621504:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573359.621507:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573359.621510:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573360.068868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573360.132842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573361.092974:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573361.156901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573362.116861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573362.180832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573363.140972:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573363.204830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573364.164822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573364.228823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573364.740826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573364.740838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573364.740841:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573364.740850:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5115 00000100:00080000:3.0:1788573364.740859:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573364.740865:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573364.740905:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864054144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573364.740963:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573364.740968:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573364.740971:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573364.740976:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5115 00000100:00080000:3.0:1788573364.740980:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573364.740984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573364.740993:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864054144:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573364.740994:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573364.741018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864054272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573364.741031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573364.741034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573364.741036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573364.741040:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5115 00000100:00080000:3.0:1788573364.741045:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573364.741050:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573364.741055:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573364.741067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864054400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573364.741076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573364.741083:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573364.741086:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573364.741087:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573364.741092:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5115 00000100:00080000:3.0:1788573364.741095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573364.741098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573364.741102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573364.741113:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864054528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573364.741120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573364.741123:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff92fc67339180 x1875449864054144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573364.741124:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573364.741128:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573364.741135:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5115 00000100:00080000:3.0:1788573364.741139:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573364.741141:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573364.741145:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573364.741147:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864054144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573364.741155:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864054656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573364.741158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573364.741168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573364.741174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864054144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573364.741194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573364.741217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573364.741227:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff92fc67339880 x1875449864054272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573364.741239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864054272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573364.741245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573364.741248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573364.741250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573364.741254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573364.741265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573364.741272:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff92fc67339c00 x1875449864054400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573364.741279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864054400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573364.741284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573364.741286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573364.741288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573364.741291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573364.741302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573364.741307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff92fc67339f80 x1875449864054528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573364.741314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864054528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573364.741324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573364.741326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573364.741328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573364.741331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573364.741341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573364.741346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff92fc6733ad80 x1875449864054656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573364.741353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864054656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573364.741358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573364.741360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573364.741362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573365.188889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573365.252804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573366.212823:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573366.276801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573367.236903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573367.300839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573368.260859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573368.324793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573369.284897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573369.348907:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573369.860841:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573369.860850:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573369.860853:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573369.860862:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5120 00000100:00080000:3.0:1788573369.860872:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573369.860878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573369.860911:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864054784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573369.860950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573369.860954:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573369.860956:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573369.860959:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5120 00000100:00080000:3.0:1788573369.860962:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573369.860964:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573369.860974:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573369.860991:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864054912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573369.860998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864054784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573369.861010:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573369.861013:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573369.861015:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573369.861019:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5120 00000100:00080000:3.0:1788573369.861023:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573369.861027:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573369.861032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573369.861046:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864055040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573369.861057:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573369.861060:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573369.861062:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573369.861064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573369.861069:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5120 00000100:00080000:3.0:1788573369.861075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573369.861080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573369.861085:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573369.861095:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864055168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573369.861102:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573369.861107:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff92fc6733bb80 x1875449864054784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573369.861107:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573369.861111:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573369.861117:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5120 00000100:00080000:3.0:1788573369.861122:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573369.861129:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573369.861133:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573369.861137:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864054784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573369.861142:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864055296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573369.861172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573369.861176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573369.861180:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864054784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573369.861200:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573369.861215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573369.861223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff92fc6733a300 x1875449864054912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573369.861231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864054912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573369.861237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573369.861239:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573369.861241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864054912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573369.861244:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573369.861254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573369.861260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff92fc6733b100 x1875449864055040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573369.861267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864055040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573369.861272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573369.861273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573369.861275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573369.861279:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573369.861290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573369.861295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff92fc67338380 x1875449864055168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573369.861303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864055168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573369.861308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573369.861309:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573369.861311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573369.861314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573369.861324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573369.861330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff92fc67338a80 x1875449864055296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573369.861337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864055296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573369.861341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573369.861343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573369.861345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573370.308851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573370.372846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573371.332970:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573371.396925:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573372.356914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573372.420827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573373.380899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573373.444847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573374.404874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573374.468811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573374.980836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573374.980847:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573374.980851:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573374.980862:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5125 00000100:00080000:3.0:1788573374.980873:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573374.980881:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573374.980919:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864055424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573374.980977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573374.980981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573374.980982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573374.980985:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5125 00000100:00080000:3.0:1788573374.980988:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573374.980992:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573374.981000:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573374.981012:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864055552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573374.981019:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573374.981022:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573374.981024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573374.981026:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5125 00000100:00080000:3.0:1788573374.981028:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573374.981030:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573374.981034:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573374.981037:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864055424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573374.981043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864055680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573374.981054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573374.981056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573374.981057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573374.981060:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5125 00000100:00080000:3.0:1788573374.981064:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573374.981068:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573374.981074:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573374.981087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864055808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573374.981097:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573374.981101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573374.981103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573374.981108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573374.981112:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5125 00000100:00080000:3.0:1788573374.981118:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573374.981122:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573374.981128:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573374.981141:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864055936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573374.981152:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff92fc67338000 x1875449864055424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573374.981169:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864055424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573374.981178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573374.981181:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573374.981184:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864055424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573374.981189:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573374.981204:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573374.981213:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff92fc67339500 x1875449864055552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573374.981221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864055552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573374.981226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573374.981228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573374.981231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573374.981249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573374.981260:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573374.981267:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff92fc6733b480 x1875449864055680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573374.981278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864055680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573374.981284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573374.981286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573374.981288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573374.981291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573374.981301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573374.981307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff92fc6733b800 x1875449864055808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573374.981314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864055808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573374.981319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573374.981321:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573374.981323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573374.981326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573374.981335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573374.981341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff92fc67338700 x1875449864055936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573374.981348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864055936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573374.981357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573374.981359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573374.981361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864055936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573375.428895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573375.492827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573376.452920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573376.516793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573377.476977:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573377.540909:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573378.500873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573378.564839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573379.524904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573379.588850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573380.100858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573380.100874:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573380.100880:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573380.100894:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5130 00000100:00080000:3.0:1788573380.100909:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573380.100920:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573380.100986:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864056064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573380.101023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573380.101028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573380.101031:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573380.101036:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5130 00000100:00080000:3.0:1788573380.101041:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573380.101046:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573380.101058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573380.101080:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864056192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573380.101088:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573380.101091:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573380.101092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573380.101096:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5130 00000100:00080000:3.0:1788573380.101099:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573380.101101:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573380.101105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573380.101114:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864056064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573380.101115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864056320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573380.101132:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573380.101135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573380.101136:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573380.101139:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5130 00000100:00080000:3.0:1788573380.101141:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573380.101143:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573380.101146:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573380.101157:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864056448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573380.101163:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573380.101165:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573380.101167:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573380.101169:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5130 00000100:00080000:3.0:1788573380.101171:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573380.101173:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573380.101176:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573380.101184:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864056576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573380.101208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573380.101266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573380/real 1788573380] req@ffff92fc6733aa00 x1875449864056064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573380.101288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864056064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573380.101301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573380.101306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573380.101311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864056064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573380.101320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573380.101346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573380.101361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573380/real 1788573380] req@ffff92fc67338e00 x1875449864056192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573380.101375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864056192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573380.101392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573380.101396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573380.101401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573380.101432:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573380.101456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573380.101470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573380/real 1788573380] req@ffff92fc6733a680 x1875449864056320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573380.101483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864056320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573380.101493:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573380.101497:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573380.101501:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573380.101508:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573380.101533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573380.101546:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573380/real 1788573380] req@ffff92fc67338700 x1875449864056448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573380.101559:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864056448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573380.101567:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573380.101571:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573380.101575:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573380.101582:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573380.101605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573380.101617:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573380/real 1788573380] req@ffff92fc6733b800 x1875449864056576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573380.101630:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864056576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573380.101638:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573380.101642:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573380.101645:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573380.548912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573380.612901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573381.572896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573381.636832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573382.596852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573382.660832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573383.620869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573383.684828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573384.644908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573384.708895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573385.220802:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573385.220811:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573385.220815:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573385.220823:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5135 00000100:00080000:0.0:1788573385.220833:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573385.220839:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573385.220871:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864056704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573385.220892:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573385.220896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573385.220897:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573385.220900:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5135 00000100:00080000:0.0:1788573385.220902:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573385.220904:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573385.220913:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573385.220923:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864056832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573385.220952:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573385.220958:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573385.220958:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864056704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573385.220960:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573385.220969:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5135 00000100:00080000:0.0:1788573385.220971:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573385.220974:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573385.220978:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573385.220988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864056960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573385.220995:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573385.220998:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573385.220999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573385.221001:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5135 00000100:00080000:0.0:1788573385.221003:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573385.221005:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573385.221010:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573385.221016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864057088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573385.221023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573385.221025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573385.221026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573385.221028:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5135 00000100:00080000:0.0:1788573385.221030:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573385.221032:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573385.221036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573385.221041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573385.221045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864057216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573385.221092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff92fc6704bb80 x1875449864056704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573385.221114:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864056704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573385.221127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573385.221131:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573385.221136:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864056704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573385.221164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573385.221198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573385.221214:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff92fc67048e00 x1875449864056832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573385.221224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864056832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573385.221241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573385.221245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573385.221251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573385.221257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573385.221290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573385.221303:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff92fc67049c00 x1875449864056960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573385.221311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864056960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573385.221317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573385.221319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573385.221321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864056960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573385.221325:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573385.221338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573385.221347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff92fc67049180 x1875449864057088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573385.221361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864057088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573385.221367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573385.221370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573385.221372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573385.221376:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573385.221389:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573385.221397:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573385/real 1788573385] req@ffff92fc67049500 x1875449864057216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573385.221407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864057216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573385.221414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573385.221416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573385.221420:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573385.668894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573385.732830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573386.692865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573386.756854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573387.716888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573387.780815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573388.740879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573388.804828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573389.764953:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573389.828820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573390.340844:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.340853:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573390.340856:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.340865:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5140 00000100:00080000:3.0:1788573390.340874:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.340880:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573390.340912:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864057344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573390.340957:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.340961:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573390.340962:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.340965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5140 00000100:00080000:3.0:1788573390.340968:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.340971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573390.340979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573390.340990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864057472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573390.340998:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.341001:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573390.341002:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.341005:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5140 00000100:00080000:3.0:1788573390.341007:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573390.341013:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864057344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573390.341013:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573390.341018:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573390.341026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864057600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573390.341034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.341037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573390.341038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.341041:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5140 00000100:00080000:3.0:1788573390.341043:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.341045:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573390.341048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573390.341055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864057728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573390.341062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573390.341065:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573390.341066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573390.341068:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5140 00000100:00080000:3.0:1788573390.341070:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573390.341073:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573390.341076:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573390.341082:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864057856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573390.341091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573390.341133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff92fc6733b480 x1875449864057344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573390.341153:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864057344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573390.341161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573390.341164:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573390.341167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864057344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573390.341172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573390.341189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573390.341197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff92fc67339500 x1875449864057472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573390.341205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864057472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573390.341211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573390.341213:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573390.341215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573390.341234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573390.341245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573390.341251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff92fc67338000 x1875449864057600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573390.341259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864057600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573390.341268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573390.341270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573390.341273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573390.341276:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573390.341286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573390.341292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff92fc67338a80 x1875449864057728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573390.341299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864057728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573390.341304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573390.341306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573390.341308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573390.341312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573390.341322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573390.341330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573390/real 1788573390] req@ffff92fc67338380 x1875449864057856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573390.341343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864057856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573390.341351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573390.341354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573390.341367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864057856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573390.789018:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573390.852866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573391.812979:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573391.876824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573392.836899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573392.900861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573393.860894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573393.924783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573394.884895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573394.948884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573395.460882:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573395.460894:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573395.460899:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573395.460910:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5146 00000100:00080000:1.0:1788573395.460922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573395.460930:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573395.461009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864057984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573395.461040:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573395.461044:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573395.461045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573395.461048:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5146 00000100:00080000:1.0:1788573395.461051:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573395.461055:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573395.461066:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573395.461077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864058112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573395.461085:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573395.461088:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573395.461090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573395.461092:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5146 00000100:00080000:1.0:1788573395.461096:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573395.461100:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573395.461108:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573395.461109:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864057984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573395.461120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864058240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573395.461130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573395.461133:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573395.461135:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573395.461138:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5146 00000100:00080000:1.0:1788573395.461142:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573395.461144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573395.461149:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573395.461160:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864058368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573395.461167:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573395.461171:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573395.461172:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573395.461175:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5146 00000100:00080000:1.0:1788573395.461179:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573395.461181:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573395.461184:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573395.461191:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864058496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573395.461223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573395.461258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573395.461273:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573395.461295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573395.908916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573395.972839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573396.932893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573396.996825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573397.956904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573398.020862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788573398.533004:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788573398.533064:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573398.533081:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573398.533134:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573398.533144:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573398.533153:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573398.533161:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788573398.533310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff92fc67289180 x1875449864058240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.533354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864058240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.533371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.533377:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573398.533383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573398.533395:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff92fc67289500 x1875449864058368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.533410:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864058368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.533419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.533423:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573398.533427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573398.533435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff92fc6728bb80 x1875449864058496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.533448:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864058496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.533457:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.533461:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573398.533466:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058496:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573398.533475:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff92fc67288700 x1875449864057984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.533489:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864057984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.533498:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.533502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573398.533506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864057984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573398.533515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573395/real 1788573398] req@ffff92fc67288a80 x1875449864058112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573398.533530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864058112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573398.533540:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573398.533544:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573398.533548:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058112:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788573398.980840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573399.044831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573400.004892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573400.068794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573400.580823:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573400.580832:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573400.580836:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573400.580845:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5151 00000100:00080000:3.0:1788573400.580854:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573400.580860:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573400.580891:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864058624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573400.580914:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573400.580918:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573400.580920:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573400.580924:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5151 00000100:00080000:3.0:1788573400.580928:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573400.580932:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573400.580943:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573400.580956:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864058752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573400.580967:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573400.580970:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864058624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573400.580970:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573400.580972:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573400.580976:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5151 00000100:00080000:3.0:1788573400.580980:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573400.580984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573400.580989:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573400.581008:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864058880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573400.581018:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573400.581021:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573400.581022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573400.581026:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5151 00000100:00080000:3.0:1788573400.581029:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573400.581031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573400.581033:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573400.581037:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573400.581047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864059008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573400.581056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573400.581070:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573400.581072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573400.581076:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5151 00000100:00080000:3.0:1788573400.581079:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573400.581083:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff92fc6733b100 x1875449864058624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573400.581084:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573400.581089:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573400.581099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864059136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573400.581103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864058624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.581121:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.581133:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573400.581138:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864058624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573400.581164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573400.581190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573400.581205:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff92fc6733a300 x1875449864058752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.581219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864058752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.581228:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.581231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573400.581234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573400.581242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573400.581263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573400.581274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff92fc6733bb80 x1875449864058880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.581286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864058880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.581294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.581297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573400.581300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864058880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573400.581307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573400.581326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573400.581337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff92fc6733ad80 x1875449864059008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.581348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864059008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.581356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.581359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573400.581361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573400.581367:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573400.581385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573400.581396:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff92fc67339f80 x1875449864059136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573400.581407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864059136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.581414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573400.581417:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573400.581420:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573401.028848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573401.092771:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573402.052900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573402.116847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573403.076834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573403.140852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573404.100894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573404.164856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573405.124884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573405.188846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573405.700855:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573405.700866:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573405.700870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573405.700881:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5156 00000100:00080000:3.0:1788573405.700892:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573405.700899:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573405.700950:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864059264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573405.700982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573405.700985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573405.700987:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573405.700990:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5156 00000100:00080000:3.0:1788573405.700993:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573405.700996:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573405.701007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573405.701018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864059392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573405.701026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573405.701029:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573405.701031:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573405.701037:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5156 00000100:00100000:0.0:1788573405.701039:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864059264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573405.701039:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573405.701045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573405.701051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573405.701057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864059520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573405.701064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573405.701066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573405.701068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573405.701070:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5156 00000100:00080000:3.0:1788573405.701072:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573405.701074:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573405.701077:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573405.701088:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864059648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573405.701097:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573405.701100:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573405.701102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573405.701106:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5156 00000100:00080000:3.0:1788573405.701111:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573405.701117:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788573405.701119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573405.701124:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573405.701139:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864059776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573405.701189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff92fc67339c00 x1875449864059264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.701212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864059264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.701233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.701239:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573405.701244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864059264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573405.701270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573405.701292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573405.701306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff92fc67339880 x1875449864059392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.701320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864059392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.701331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.701335:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573405.701339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573405.701346:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573405.701364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573405.701372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff92fc67339180 x1875449864059520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.701386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864059520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.701392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.701394:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573405.701396:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573405.701400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573405.701413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573405.701420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff92fc67339f80 x1875449864059648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.701427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864059648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.701433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.701435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573405.701437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573405.701441:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573405.701451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573405.701457:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff92fc6733ad80 x1875449864059776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573405.701464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864059776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573405.701474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573405.701476:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573405.701479:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864059776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573406.148912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573406.212887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573407.172867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573407.236810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573408.196884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573408.260841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573409.220868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573409.284785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573410.244960:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573410.308813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573410.820846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573410.820855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573410.820858:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573410.820867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5161 00000100:00080000:3.0:1788573410.820876:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573410.820883:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573410.820915:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864059904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573410.820939:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573410.820944:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573410.820946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573410.820951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5161 00000100:00080000:3.0:1788573410.820955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573410.820959:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573410.820979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573410.820991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864059904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573410.820992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864060032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573410.821003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573410.821006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573410.821009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573410.821012:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5161 00000100:00080000:3.0:1788573410.821016:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573410.821020:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573410.821025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573410.821034:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864060160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573410.821043:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573410.821046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573410.821047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573410.821051:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5161 00000100:00080000:3.0:1788573410.821054:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573410.821056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573410.821058:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573410.821062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573410.821071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864060288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573410.821080:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573410.821083:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573410.821085:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573410.821088:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5161 00000100:00080000:3.0:1788573410.821092:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573410.821095:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573410.821097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff92fc6733bb80 x1875449864059904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573410.821100:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573410.821109:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864060416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573410.821116:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864059904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573410.821126:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573410.821129:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573410.821133:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864059904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573410.821156:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573410.821179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573410.821190:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff92fc6733a300 x1875449864060032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573410.821204:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864060032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573410.821210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573410.821213:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573410.821215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573410.821219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573410.821232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573410.821238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff92fc6733b100 x1875449864060160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573410.821265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864060160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573410.821271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573410.821274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573410.821276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573410.821281:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573410.821296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573410.821303:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff92fc67338380 x1875449864060288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573410.821311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864060288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573410.821320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573410.821322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573410.821324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573410.821328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573410.821347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573410.821354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573410] req@ffff92fc67338a80 x1875449864060416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573410.821361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864060416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573410.821366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573410.821368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573410.821370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573411.268919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573411.332846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573412.292926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573412.356801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573413.316873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573413.380827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573414.340865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573414.404808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573415.364872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573415.428795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573415.940819:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.940827:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573415.940830:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.940847:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5166 00000100:00080000:3.0:1788573415.940856:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.940863:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573415.940895:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864060544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.940917:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.940925:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573415.940927:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.940930:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5166 00000100:00080000:3.0:1788573415.940932:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.940935:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573415.940943:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573415.940953:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864060672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.940962:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.940965:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573415.940968:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864060544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573415.940968:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.940976:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5166 00000100:00080000:3.0:1788573415.940980:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.940983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573415.940985:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573415.940992:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864060800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.940999:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.941002:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573415.941003:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.941005:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5166 00000100:00080000:3.0:1788573415.941007:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573415.941013:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573415.941017:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573415.941024:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864060928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573415.941030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573415.941032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573415.941034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573415.941036:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5166 00000100:00080000:3.0:1788573415.941038:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573415.941042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573415.941046:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573415.941050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573415.941058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864061056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573415.941083:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff92fc67338000 x1875449864060544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.941098:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864060544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.941107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.941110:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573415.941113:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864060544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573415.941131:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573415.941145:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573415.941154:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff92fc67339500 x1875449864060672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.941162:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864060672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.941168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.941170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573415.941172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573415.941176:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573415.941186:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573415.941193:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff92fc6733b480 x1875449864060800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.941200:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864060800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.941210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.941212:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573415.941215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573415.941218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573415.941228:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573415.941234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff92fc6733b800 x1875449864060928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.941241:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864060928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.941246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.941248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573415.941250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864060928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573415.941254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573415.941264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573415.941269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff92fc67338700 x1875449864061056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573415.941276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864061056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573415.941281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573415.941282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573415.941285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573416.388881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573416.452856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573417.412859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573417.476848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573418.436844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573418.500826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573419.460856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573419.524819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573420.484845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573420.548819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573421.060829:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573421.060840:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573421.060845:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573421.060855:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5171 00000100:00080000:3.0:1788573421.060867:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573421.060875:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573421.060916:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864061184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573421.060946:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573421.060951:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573421.060953:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573421.060956:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5171 00000100:00080000:3.0:1788573421.060960:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573421.060963:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573421.060972:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573421.060983:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864061312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573421.060992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573421.060996:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573421.060997:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864061184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573421.060998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573421.061007:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5171 00000100:00080000:3.0:1788573421.061009:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573421.061012:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573421.061015:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573421.061024:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864061440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573421.061031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573421.061034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573421.061035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573421.061039:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5171 00000100:00080000:3.0:1788573421.061043:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573421.061045:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573421.061048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573421.061056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864061568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573421.061062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573421.061065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573421.061066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573421.061068:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5171 00000100:00080000:3.0:1788573421.061070:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573421.061072:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573421.061075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573421.061080:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573421.061095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864061696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573421.061121:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573421/real 1788573421] req@ffff92fc6733a680 x1875449864061184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573421.061137:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864061184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573421.061147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573421.061153:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573421.061158:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864061184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573421.061194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573421.061222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573421.061233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573421/real 1788573421] req@ffff92fc67338e00 x1875449864061312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573421.061246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864061312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573421.061255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573421.061259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573421.061263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573421.061277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573421.061296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573421.061309:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573421/real 1788573421] req@ffff92fc6733aa00 x1875449864061440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573421.061325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864061440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573421.061331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573421.061334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573421.061337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573421.061341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573421.061355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573421.061362:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573421/real 1788573421] req@ffff92fc67338700 x1875449864061568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573421.061369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864061568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573421.061375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573421.061378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573421.061380:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573421.061383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573421.061396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573421.061406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573421/real 1788573421] req@ffff92fc6733b800 x1875449864061696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573421.061428:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864061696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573421.061450:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573421.061454:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573421.061457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573421.508851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573421.572832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573422.532898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573422.596868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573423.556856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573423.620833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573424.580901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573424.644808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573425.604939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573425.668866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573426.180857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.180870:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573426.180875:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.180888:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5176 00000100:00080000:0.0:1788573426.180900:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.180908:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573426.180954:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864061824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.180988:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.180993:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573426.180994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.180997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5176 00000100:00080000:0.0:1788573426.181000:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.181002:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573426.181012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573426.181023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864061952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.181031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.181034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573426.181035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.181038:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5176 00000100:00080000:0.0:1788573426.181040:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.181042:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573426.181046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573426.181052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864061824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573426.181057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864062080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.181065:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.181068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573426.181069:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573426.181072:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5176 00000100:00080000:0.0:1788573426.181075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.181084:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573426.181092:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573426.181107:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864062208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573426.181118:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573426.181122:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573426.181124:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788573426.181130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573426.181134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5176 00000100:00080000:0.0:1788573426.181140:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573426.181145:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573426.181152:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573426.181169:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864062336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573426.181182:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff92fc6704b100 x1875449864061824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573426.181199:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864061824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573426.181210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573426.181214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573426.181217:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864061824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573426.181223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573426.181241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573426.181251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff92fc6704ad80 x1875449864061952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573426.181259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864061952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573426.181265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573426.181267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573426.181269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864061952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573426.181294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573426.181305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573426.181311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff92fc6704a680 x1875449864062080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573426.181318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864062080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573426.181324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573426.181326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573426.181328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573426.181331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573426.181342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573426.181348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff92fc6704aa00 x1875449864062208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573426.181360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864062208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573426.181365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573426.181367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573426.181369:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573426.181372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573426.181382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573426.181387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573426/real 1788573426] req@ffff92fc6704a300 x1875449864062336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573426.181394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864062336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573426.181398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573426.181400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573426.181402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573426.628966:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573426.692850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573427.652904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573427.716888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573428.676897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573428.740797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573429.700945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573429.764946:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573430.724890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573430.788830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573431.300844:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573431.300853:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573431.300856:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573431.300865:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5181 00000100:00080000:0.0:1788573431.300874:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573431.300879:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573431.300911:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864062464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573431.300934:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573431.300937:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573431.300939:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573431.300987:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5181 00000100:00080000:0.0:1788573431.300990:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573431.300996:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573431.301005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573431.301015:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864062592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573431.301024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573431.301026:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573431.301028:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573431.301034:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5181 00000100:00100000:1.0:1788573431.301039:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864062464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573431.301040:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573431.301047:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573431.301063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573431.301072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864062720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573431.301080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573431.301084:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573431.301086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573431.301092:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5181 00000100:00080000:0.0:1788573431.301096:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573431.301101:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573431.301107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573431.301122:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864062848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573431.301133:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573431.301136:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573431.301138:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573431.301142:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5181 00000800:00000100:1.0:1788573431.301148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573431.301149:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573431.301162:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573431.301168:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573431.301185:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864062976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573431.301238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff92fc67049880 x1875449864062464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573431.301369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864062464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573431.301384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573431.301390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573431.301395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864062464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573431.301445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573431.301480:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573431.301498:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff92fc67048000 x1875449864062592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573431.301512:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864062592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573431.301522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573431.301526:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573431.301530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573431.301538:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573431.301562:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573431.301575:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff92fc67049f80 x1875449864062720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573431.301589:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864062720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573431.301605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573431.301610:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573431.301613:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573431.301654:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573431.301678:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573431.301690:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff92fc67048700 x1875449864062848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573431.301703:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864062848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573431.301712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573431.301716:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573431.301759:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573431.301767:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573431.301791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573431.301804:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573431/real 1788573431] req@ffff92fc67048380 x1875449864062976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573431.301818:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864062976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573431.301826:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573431.301830:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573431.301835:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864062976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573431.748839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573431.812840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573432.772924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573432.836785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573433.796865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573433.860835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573434.820937:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573434.884825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573435.844844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573435.908862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573436.420901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573436.420913:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573436.420918:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573436.420928:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5187 00000100:00080000:3.0:1788573436.420941:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573436.420949:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573436.420994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864063104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573436.421028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573436.421033:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573436.421035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573436.421040:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5187 00000100:00080000:3.0:1788573436.421044:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573436.421049:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573436.421061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573436.421076:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864063232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573436.421086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573436.421090:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864063104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573436.421091:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573436.421093:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573436.421100:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5187 00000100:00080000:3.0:1788573436.421104:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573436.421107:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573436.421111:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573436.421122:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864063360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573436.421129:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573436.421133:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573436.421135:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573436.421139:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5187 00000100:00080000:3.0:1788573436.421145:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573436.421150:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573436.421156:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573436.421162:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573436.421170:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864063488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573436.421182:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573436.421185:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573436.421186:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573436.421190:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5187 00000100:00080000:3.0:1788573436.421194:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573436.421196:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573436.421201:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573436.421211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff92fc6733b480 x1875449864063104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573436.421212:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864063616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573436.421251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864063104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573436.421266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573436.421270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573436.421274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864063104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573436.421300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573436.421319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573436.421329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff92fc67339500 x1875449864063232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573436.421338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864063232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573436.421345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573436.421347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573436.421349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573436.421353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573436.421366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573436.421373:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff92fc67338000 x1875449864063360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573436.421380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864063360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573436.421386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573436.421388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573436.421389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573436.421393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573436.421405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573436.421411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff92fc67338a80 x1875449864063488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573436.421418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864063488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573436.421423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573436.421425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573436.421427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573436.421433:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573436.421452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573436.421464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573436/real 1788573436] req@ffff92fc67338380 x1875449864063616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573436.421485:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864063616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573436.421491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573436.421494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573436.421496:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573436.868944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573436.932887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573437.892881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573437.956819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573438.916910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573438.980862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573439.940859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573440.004913:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573440.964917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573441.028853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573441.540837:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573441.540845:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573441.540848:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573441.540857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5192 00000100:00080000:3.0:1788573441.540866:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573441.540873:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573441.540905:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864063744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573441.540933:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573441.540938:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573441.540939:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573441.540942:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5192 00000100:00080000:3.0:1788573441.540944:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573441.540947:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573441.540955:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573441.540968:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864063872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573441.540980:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573441.540985:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864063744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573441.540986:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573441.540991:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573441.540997:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5192 00000100:00080000:3.0:1788573441.541002:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573441.541007:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573441.541012:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573441.541026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864064000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573441.541033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573441.541036:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573441.541037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573441.541039:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5192 00000100:00080000:3.0:1788573441.541043:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573441.541047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573441.541050:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573441.541056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573441.541065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864064128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573441.541072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573441.541074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573441.541075:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573441.541078:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5192 00000100:00080000:3.0:1788573441.541081:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573441.541084:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573441.541087:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff92fc6733b100 x1875449864063744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573441.541089:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573441.541101:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864064256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573441.541108:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864063744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573441.541117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573441.541120:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573441.541124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864063744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573441.541143:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573441.541158:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573441.541167:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff92fc6733a300 x1875449864063872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573441.541180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864063872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573441.541186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573441.541188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573441.541190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864063872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573441.541194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573441.541204:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573441.541220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff92fc6733bb80 x1875449864064000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573441.541228:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864064000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573441.541233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573441.541236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573441.541238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573441.541241:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573441.541253:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573441.541258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff92fc6733ad80 x1875449864064128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573441.541265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864064128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573441.541275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573441.541284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573441.541286:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573441.541290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573441.541301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573441.541308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573441/real 1788573441] req@ffff92fc67339f80 x1875449864064256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573441.541315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864064256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573441.541320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573441.541322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573441.541324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573441.988905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573442.052857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573443.012908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573443.076795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573444.036920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573444.100825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573445.060997:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573445.124859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573446.084853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573446.148832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573446.660883:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573446.660896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573446.660923:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573446.660935:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5197 00000100:00080000:3.0:1788573446.660947:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573446.660956:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573446.661009:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864064384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573446.661053:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573446.661062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573446.661065:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573446.661069:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5197 00000100:00080000:3.0:1788573446.661075:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573446.661079:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573446.661092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573446.661108:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864064512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573446.661120:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864064384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573446.661123:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573446.661130:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573446.661132:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573446.661138:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5197 00000100:00080000:3.0:1788573446.661142:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573446.661147:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573446.661152:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573446.661167:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864064640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573446.661177:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573446.661181:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573446.661183:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573446.661187:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5197 00000100:00080000:3.0:1788573446.661192:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573446.661196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573446.661200:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573446.661208:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573446.661224:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864064768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573446.661240:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573446.661244:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573446.661247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff92fc67339180 x1875449864064384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573446.661248:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573446.661258:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5197 00000100:00080000:3.0:1788573446.661266:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573446.661272:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573446.661275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864064384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573446.661280:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573446.661289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573446.661294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573446.661299:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864064896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573446.661302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864064384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573446.661332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573446.661350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573446.661360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff92fc67339880 x1875449864064512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573446.661368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864064512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573446.661375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573446.661378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573446.661380:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573446.661384:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573446.661395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573446.661405:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff92fc67339c00 x1875449864064640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573446.661417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864064640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573446.661425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573446.661429:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573446.661432:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573446.661439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573446.661461:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573446.661470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff92fc67339f80 x1875449864064768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573446.661478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864064768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573446.661483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573446.661486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573446.661488:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573446.661492:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573446.661505:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573446.661515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff92fc6733ad80 x1875449864064896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573446.661527:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864064896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573446.661536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573446.661540:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573446.661544:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864064896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573447.108896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573447.172989:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573448.132919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573448.196814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573449.156936:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573449.220829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573450.181037:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573450.244812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573451.204944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573451.268862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573451.780866:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.780880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573451.780886:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.780898:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5202 00000100:00080000:2.0:1788573451.780911:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.780921:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573451.780970:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864065024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573451.781021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.781026:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573451.781027:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.781031:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5202 00000100:00080000:2.0:1788573451.781036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.781041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573451.781052:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573451.781071:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864065152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573451.781082:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864065024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573451.781110:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.781115:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573451.781118:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.781124:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5202 00000100:00080000:2.0:1788573451.781130:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.781136:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573451.781143:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573451.781159:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864065280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573451.781160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573451.781178:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.781189:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573451.781192:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.781200:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5202 00000100:00080000:2.0:1788573451.781206:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573451.781210:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff92fc4617ca80 x1875449864065024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573451.781216:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573451.781233:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573451.781244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864065024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573451.781248:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864065408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573451.781289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573451.781290:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.781301:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573451.781306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788573451.781309:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573451.781317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864065024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573451.781320:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5202 00000100:00080000:2.0:1788573451.781331:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.781335:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573451.781339:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573451.781348:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864065536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573451.781352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573451.781386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573451.781399:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff92fc4617dc00 x1875449864065152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573451.781409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864065152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573451.781416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573451.781419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573451.781421:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573451.781426:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573451.781437:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573451.781452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff92fc4617e680 x1875449864065280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573451.781460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864065280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573451.781470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573451.781472:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573451.781475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573451.781478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573451.781491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573451.781501:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff92fc4617d180 x1875449864065408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573451.781513:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864065408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573451.781521:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573451.781524:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573451.781528:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573451.781548:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573451.781566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573451.781574:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff92fc4617c380 x1875449864065536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573451.781582:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864065536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573451.781587:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573451.781589:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573451.781592:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573452.228897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573452.292809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573453.252966:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573453.316832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573454.276859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573454.340821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573455.300892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573455.364844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573456.324821:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573456.388830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573456.900806:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.900813:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573456.900817:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.900828:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5207 00000100:00080000:2.0:1788573456.900839:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.900848:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573456.900885:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864065664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.900918:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.900925:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573456.900928:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.900933:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5207 00000100:00080000:2.0:1788573456.900939:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.900944:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573456.900952:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864065664:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788573456.900954:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573456.900972:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864065792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.900983:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.900986:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573456.900988:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.900991:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5207 00000100:00080000:2.0:1788573456.900995:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.900998:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573456.901001:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573456.901010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864065920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.901017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573456.901018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573456.901023:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573456.901025:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.901029:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5207 00000100:00080000:2.0:1788573456.901039:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.901041:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573456.901046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573456.901056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864066048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573456.901060:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff92fc4617fb80 x1875449864065664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573456.901076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.901086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573456.901106:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.901112:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5207 00000100:00080000:2.0:1788573456.901118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.901123:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573456.901127:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864065664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573456.901132:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573456.901138:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.901143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573456.901146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864065664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573456.901152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864066176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573456.901170:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573456.901199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573456.901209:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff92fc4617c000 x1875449864065792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.901217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864065792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.901223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.901225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573456.901227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573456.901231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573456.901243:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573456.901252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff92fc4617d880 x1875449864065920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.901260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864065920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.901269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.901272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573456.901274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864065920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573456.901278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573456.901289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573456.901298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff92fc4617e300 x1875449864066048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.901307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864066048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.901313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.901316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573456.901318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573456.901324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573456.901336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573456.901344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff92fc4617ea00 x1875449864066176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573456.901354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864066176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573456.901361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573456.901364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573456.901367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573457.348854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573457.412822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573458.372852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573458.436826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573459.396864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573459.460818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573460.420868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573460.484824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573461.444879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573461.508835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573462.020898:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573462.020906:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573462.020910:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573462.020918:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5212 00000100:00080000:3.0:1788573462.020932:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573462.020939:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573462.020972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864066304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573462.020994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573462.020997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573462.020999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573462.021002:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5212 00000100:00080000:3.0:1788573462.021004:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573462.021006:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573462.021015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573462.021023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864066432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573462.021031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573462.021033:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573462.021035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573462.021037:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5212 00000100:00080000:3.0:1788573462.021039:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573462.021041:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573462.021044:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573462.021054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864066560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573462.021055:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864066304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573462.021064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573462.021067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573462.021069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573462.021073:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5212 00000100:00080000:3.0:1788573462.021080:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573462.021083:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573462.021117:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573462.021140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864066688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573462.021152:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573462.021155:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573462.021158:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573462.021162:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5212 00000100:00080000:3.0:1788573462.021167:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573462.021171:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573462.021177:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573462.021191:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864066816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573462.021208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573462.021247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573462.021275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573462.021301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573462.468883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573462.532835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573463.492904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573463.556825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573464.516853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573464.580863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788573465.092954:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788573465.092991:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573465.093008:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573465.093044:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573465.093056:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573465.093064:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573465.093075:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788573465.093098:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573462/real 1788573465] req@ffff92fc6733b100 x1875449864066560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573465.093127:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864066560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573465.093141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573465.093146:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573465.093152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573465.093161:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573462/real 1788573465] req@ffff92fc67338380 x1875449864066688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573465.093175:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864066688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573465.093182:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573465.093185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573465.093189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573465.093195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573462/real 1788573465] req@ffff92fc67338a80 x1875449864066816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573465.093206:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864066816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573465.093213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573465.093231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573465.093234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066816:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573465.093241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573462/real 1788573465] req@ffff92fc6733bb80 x1875449864066304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573465.093254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864066304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573465.093261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573465.093264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573465.093268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864066304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573465.093274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573462/real 1788573465] req@ffff92fc6733a300 x1875449864066432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573465.093284:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864066432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573465.093292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573465.093295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573465.093299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864066432:192.168.202.132@tcp:38:kworker.0 00010000:00010000:1.0:1788573465.540906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573465.604810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573466.564880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573466.628824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573467.140814:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573467.140827:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573467.140831:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573467.140850:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5217 00000100:00080000:3.0:1788573467.140861:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573467.140870:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573467.140905:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864066944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573467.140941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573467.140945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573467.140947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573467.140951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5217 00000100:00080000:3.0:1788573467.140955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573467.140959:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573467.140970:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573467.140987:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864067072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573467.141000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864066944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573467.141012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573467.141017:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573467.141020:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573467.141025:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5217 00000100:00080000:3.0:1788573467.141029:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573467.141033:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573467.141039:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573467.141053:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864067200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573467.141063:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573467.141067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573467.141069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573467.141072:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5217 00000100:00080000:3.0:1788573467.141076:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573467.141093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573467.141098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573467.141106:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573467.141119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864067328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573467.141130:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573467.141134:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573467.141136:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573467.141142:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5217 00000100:00100000:0.0:1788573467.141143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573467/real 1788573467] req@ffff92fc67338000 x1875449864066944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573467.141149:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573467.141156:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573467.141165:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573467.141175:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864066944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573467.141180:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864067456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573467.141201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573467.141208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573467.141212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864066944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573467.141240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573467.141267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573467.141281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573467/real 1788573467] req@ffff92fc67339500 x1875449864067072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573467.141293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864067072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573467.141303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573467.141306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573467.141310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573467.141317:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573467.141338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573467.141351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573467/real 1788573467] req@ffff92fc6733b480 x1875449864067200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573467.141363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864067200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573467.141370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573467.141373:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573467.141377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573467.141384:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573467.141404:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573467.141416:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573467/real 1788573467] req@ffff92fc6733b800 x1875449864067328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573467.141430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864067328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573467.141439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573467.141442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573467.141446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573467.141452:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573467.141475:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573467.141487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573467/real 1788573467] req@ffff92fc67338700 x1875449864067456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573467.141500:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864067456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573467.141509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573467.141512:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573467.141516:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573467.588947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573467.652812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573468.612856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573468.676843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573469.636907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573469.700792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573470.660834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573470.724873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573471.684910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573471.748795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573472.260810:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573472.260823:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573472.260828:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573472.260841:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5222 00000100:00080000:3.0:1788573472.260854:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573472.260863:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573472.260911:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864067584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573472.260938:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573472.260941:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573472.260942:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573472.260945:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5222 00000100:00080000:3.0:1788573472.260948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573472.260951:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573472.260960:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573472.260971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864067712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573472.260982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573472.260990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573472.260992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573472.260997:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5222 00000100:00080000:3.0:1788573472.261001:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573472.261008:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573472.261010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864067584:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573472.261015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573472.261030:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864067840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573472.261042:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573472.261046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573472.261048:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573472.261054:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5222 00000100:00080000:3.0:1788573472.261058:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573472.261063:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573472.261068:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573472.261100:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864067968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573472.261108:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573472.261112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573472.261113:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573472.261117:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573472.261121:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5222 00000100:00080000:3.0:1788573472.261125:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573472.261128:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573472.261133:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573472.261144:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864068096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573472.261184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff92fc6733aa00 x1875449864067584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573472.261217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864067584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573472.261229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573472.261234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573472.261239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864067584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573472.261247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573472.261273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573472.261285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff92fc67338e00 x1875449864067712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573472.261297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864067712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573472.261306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573472.261310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573472.261314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573472.261341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573472.261359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573472.261370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff92fc6733a680 x1875449864067840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573472.261382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864067840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573472.261397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573472.261400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573472.261404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573472.261410:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573472.261428:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573472.261438:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff92fc67338700 x1875449864067968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573472.261450:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864067968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573472.261458:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573472.261462:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573472.261465:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864067968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573472.261471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573472.261488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573472.261498:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573472/real 1788573472] req@ffff92fc6733b800 x1875449864068096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573472.261511:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864068096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573472.261520:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573472.261524:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573472.261528:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573472.708840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573472.772838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573473.732928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573473.796833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573474.756889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573474.820873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573475.780908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573475.844838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573476.804865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573476.868848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573477.380826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573477.380838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573477.380842:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573477.380853:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5227 00000100:00080000:3.0:1788573477.380863:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573477.380870:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573477.380904:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864068224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573477.380930:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573477.380935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573477.380937:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573477.380942:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5227 00000100:00080000:3.0:1788573477.380945:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573477.380950:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573477.380960:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573477.380976:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864068352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573477.380988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864068224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573477.381000:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573477.381003:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573477.381005:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573477.381008:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5227 00000100:00080000:3.0:1788573477.381013:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573477.381016:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573477.381021:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573477.381032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864068480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573477.381039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573477.381041:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573477.381042:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573477.381045:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5227 00000100:00080000:3.0:1788573477.381047:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573477.381049:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573477.381053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573477.381057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573477.381077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864068608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573477.381085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573477.381088:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573477.381089:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573477.381092:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5227 00000100:00080000:3.0:1788573477.381095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573477.381098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573477.381102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573477.381111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864068736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573477.381130:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff92fc6733b480 x1875449864068224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573477.381152:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864068224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573477.381165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573477.381170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573477.381175:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864068224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573477.381199:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573477.381225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573477.381238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff92fc67339500 x1875449864068352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573477.381250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864068352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573477.381266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573477.381270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573477.381273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573477.381280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573477.381298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573477.381308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff92fc67338000 x1875449864068480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573477.381320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864068480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573477.381329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573477.381332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573477.381336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573477.381341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573477.381370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573477.381385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff92fc6733a300 x1875449864068608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573477.381398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864068608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573477.381406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573477.381409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573477.381413:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573477.381419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573477.381438:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573477.381448:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573477/real 1788573477] req@ffff92fc6733bb80 x1875449864068736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573477.381460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864068736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573477.381468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573477.381471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573477.381474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573477.828890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573477.892820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573478.852864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573478.916835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573479.876899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573479.940800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573480.900880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573480.964855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573481.924954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573481.988842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573482.500806:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.500814:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573482.500818:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.500827:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5233 00000100:00080000:0.0:1788573482.500836:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.500842:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573482.500873:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864068864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.500895:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.500898:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573482.500899:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.500902:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5233 00000100:00080000:0.0:1788573482.500904:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.500906:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573482.500913:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573482.500923:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864068992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.500930:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.500933:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573482.500934:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.500937:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5233 00000100:00080000:0.0:1788573482.500939:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573482.500943:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864068864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573482.500944:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573482.500951:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573482.500958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864069120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.500965:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.500968:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573482.500969:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.500972:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5233 00000100:00080000:0.0:1788573482.500974:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.500977:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573482.500981:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573482.500988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864069248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573482.500994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573482.500996:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573482.500997:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573482.500999:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5233 00000100:00080000:0.0:1788573482.501002:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573482.501004:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573482.501007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573482.501013:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864069376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573482.501018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573482.501088:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff92fc67048a80 x1875449864068864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573482.501104:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864068864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573482.501113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573482.501116:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573482.501119:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864068864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573482.501124:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573482.501149:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573482.501159:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff92fc67048380 x1875449864068992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573482.501167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864068992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573482.501173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573482.501175:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573482.501178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864068992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573482.501197:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573482.501209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573482.501216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff92fc67048700 x1875449864069120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573482.501224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864069120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573482.501229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573482.501232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573482.501234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573482.501238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573482.501250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573482.501256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff92fc67049f80 x1875449864069248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573482.501263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864069248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573482.501268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573482.501270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573482.501272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573482.501275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573482.501285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573482.501291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573482/real 1788573482] req@ffff92fc67048000 x1875449864069376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573482.501298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864069376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573482.501303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573482.501305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573482.501307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573482.948896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573483.012819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573483.972881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573484.036822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573484.996871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573485.060820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573486.020906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573486.084833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573487.044881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573487.108807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573487.620868:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573487.620877:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573487.620881:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573487.620890:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5238 00000100:00080000:2.0:1788573487.620899:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573487.620904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573487.620943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864069504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573487.620970:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573487.620975:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573487.620977:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573487.620980:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5238 00000100:00080000:2.0:1788573487.620985:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573487.620989:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573487.620999:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573487.621015:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864069632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573487.621027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573487.621031:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573487.621034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573487.621037:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864069504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573487.621054:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5238 00000100:00080000:2.0:1788573487.621059:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573487.621063:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573487.621069:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573487.621086:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864069760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573487.621094:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573487.621096:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573487.621098:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573487.621101:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5238 00000100:00080000:2.0:1788573487.621104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573487.621107:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573487.621111:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573487.621119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864069888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573487.621130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573487.621132:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573487.621134:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573487.621136:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5238 00000100:00080000:2.0:1788573487.621138:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573487.621140:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573487.621143:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573487.621148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573487.621151:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864070016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573487.621198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff92fc4617d500 x1875449864069504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573487.621220:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864069504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573487.621232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573487.621237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573487.621242:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864069504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573487.621266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573487.621292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573487.621306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff92fc4617ce00 x1875449864069632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573487.621320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864069632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573487.621329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573487.621333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573487.621336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573487.621343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573487.621364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573487.621376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff92fc4617ea00 x1875449864069760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573487.621396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864069760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573487.621405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573487.621409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573487.621412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573487.621418:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573487.621439:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573487.621451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff92fc4617e300 x1875449864069888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573487.621463:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864069888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573487.621471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573487.621474:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573487.621478:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864069888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573487.621484:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573487.621501:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573487.621511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff92fc4617d880 x1875449864070016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573487.621522:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864070016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573487.621537:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573487.621541:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573487.621544:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573488.068902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573488.132842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573489.092807:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573489.156811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573490.116942:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573490.180863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573491.140876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573491.204804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573492.164909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573492.228861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573492.740869:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.740881:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573492.740886:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.740897:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5243 00000100:00080000:0.0:1788573492.740909:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.740918:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573492.740960:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864070144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573492.740991:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.740994:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573492.740996:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.740999:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5243 00000100:00080000:0.0:1788573492.741002:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.741005:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573492.741016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573492.741041:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864070144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573492.741048:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864070272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573492.741063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.741067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573492.741070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.741076:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5243 00000100:00080000:0.0:1788573492.741081:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.741088:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573492.741096:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573492.741116:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864070400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573492.741121:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573492.741134:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.741138:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573492.741141:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.741147:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5243 00000100:00080000:0.0:1788573492.741152:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.741157:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573492.741164:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573492.741168:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff92fc67049880 x1875449864070144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573492.741182:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864070528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573492.741196:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573492.741197:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864070144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573492.741199:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573492.741204:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.741213:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5243 00000100:00080000:2.0:1788573492.741217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.741222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788573492.741224:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.741231:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573492.741231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864070144:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788573492.741241:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573492.741253:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864070656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573492.741265:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573492.741288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573492.741301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff92fc6704a300 x1875449864070272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.741311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864070272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.741317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.741320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573492.741322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573492.741326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573492.741337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573492.741344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff92fc6704aa00 x1875449864070400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.741351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864070400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.741356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.741358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573492.741360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573492.741364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573492.741375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573492.741381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff92fc6704a680 x1875449864070528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.741389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864070528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.741399:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.741401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573492.741403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573492.741407:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573492.741418:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573492.741423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff92fc6704ad80 x1875449864070656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.741431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864070656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.741436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.741439:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573492.741441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573493.188950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573493.252862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573494.212997:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573494.276844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573495.236892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573495.300887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573496.260857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573496.324799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573497.284837:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573497.348836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573497.860811:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.860820:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573497.860824:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.860833:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5248 00000100:00080000:3.0:1788573497.860842:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.860848:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573497.860897:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864070784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.860923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.860926:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573497.860927:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.860931:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5248 00000100:00080000:3.0:1788573497.860935:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.860939:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573497.860948:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573497.860957:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864070784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573497.860963:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864070912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.860972:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.860975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573497.860976:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.860979:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5248 00000100:00080000:3.0:1788573497.860981:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.860983:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573497.860988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573497.860996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864071040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.861003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.861005:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573497.861007:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.861010:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5248 00000100:00080000:3.0:1788573497.861012:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.861014:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573497.861017:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573497.861035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573497.861040:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864071168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.861048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.861050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573497.861051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.861054:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5248 00000100:00080000:3.0:1788573497.861058:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.861061:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573497.861064:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573497.861073:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864071296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573497.861081:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff92fc67338a80 x1875449864070784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.861096:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864070784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.861105:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.861109:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573497.861113:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864070784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573497.861139:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573497.861165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573497.861179:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff92fc67338380 x1875449864070912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.861188:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864070912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.861194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.861197:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573497.861199:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864070912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573497.861204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573497.861218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573497.861224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff92fc6733b100 x1875449864071040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.861232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864071040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.861237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.861239:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573497.861241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573497.861244:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573497.861256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573497.861262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff92fc6733ad80 x1875449864071168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.861269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864071168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.861275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.861276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573497.861278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573497.861282:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573497.861292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573497.861297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff92fc67339f80 x1875449864071296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.861305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864071296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.861310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.861311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573497.861313:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573498.308841:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573498.372827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573499.332869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573499.396827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573500.356857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573500.420839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573501.380863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573501.444846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573502.404961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573502.468850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573502.980904:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573502.980916:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573502.980922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573502.980933:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5253 00000100:00080000:0.0:1788573502.980945:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573502.980953:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573502.980998:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864071424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573502.981054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573502.981057:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573502.981058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573502.981062:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5253 00000100:00080000:0.0:1788573502.981065:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573502.981068:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573502.981080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573502.981091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864071552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573502.981099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573502.981101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573502.981103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573502.981106:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5253 00000100:00080000:0.0:1788573502.981108:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573502.981111:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573502.981115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573502.981122:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864071680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573502.981131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573502.981133:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573502.981134:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573502.981137:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5253 00000100:00080000:0.0:1788573502.981139:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573502.981143:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573502.981151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573502.981153:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864071424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573502.981162:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864071808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573502.981171:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573502.981174:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573502.981177:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573502.981182:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5253 00000100:00080000:0.0:1788573502.981184:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573502.981187:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573502.981190:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573502.981200:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864071936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573502.981226:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573502.981266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff92fc6704b100 x1875449864071424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.981281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864071424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.981295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.981298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573502.981301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864071424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573502.981307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573502.981323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573502.981332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff92fc67049500 x1875449864071552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.981340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864071552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.981346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.981348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573502.981351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573502.981373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573502.981384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573502.981391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff92fc67049180 x1875449864071680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.981398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864071680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.981404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.981406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573502.981408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573502.981412:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573502.981423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573502.981430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff92fc67049c00 x1875449864071808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.981437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864071808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.981443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.981445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573502.981447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573502.981450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573502.981461:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573502.981467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff92fc67048e00 x1875449864071936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.981474:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864071936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.981479:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.981481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573502.981483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864071936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573503.428894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573503.492882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573504.452959:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573504.516808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573505.476931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573505.540926:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573506.500930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573506.564859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573507.524880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573507.588868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573508.100789:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573508.100804:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573508.100810:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573508.100823:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5258 00000100:00080000:0.0:1788573508.100836:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573508.100856:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573508.100902:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864072064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573508.100926:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573508.100929:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573508.100931:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573508.100935:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5258 00000100:00080000:0.0:1788573508.100938:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573508.100942:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573508.100952:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573508.100963:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864072064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573508.100966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864072192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573508.100979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573508.100982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573508.100983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573508.100985:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5258 00000100:00080000:0.0:1788573508.100987:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573508.100989:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573508.100992:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573508.101016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864072320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573508.101024:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573508.101026:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573508.101028:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573508.101030:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5258 00000100:00080000:0.0:1788573508.101034:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573508.101037:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573508.101042:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573508.101056:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864072448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573508.101067:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573508.101071:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573508.101073:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573508.101080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5258 00000800:00000100:2.0:1788573508.101081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573508.101085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573508.101092:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573508.101098:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573508.101113:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864072576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573508.101151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573508/real 1788573508] req@ffff92fc6704bb80 x1875449864072064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573508.101170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864072064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573508.101179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573508.101183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573508.101186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864072064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573508.101212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573508.101250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573508.101262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573508/real 1788573508] req@ffff92fc67048e00 x1875449864072192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573508.101287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864072192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573508.101294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573508.101296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573508.101299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573508.101304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573508.101331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573508.101339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573508/real 1788573508] req@ffff92fc67049c00 x1875449864072320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573508.101348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864072320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573508.101359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573508.101362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573508.101364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573508.101367:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573508.101380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573508.101388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573508/real 1788573508] req@ffff92fc67049180 x1875449864072448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573508.101397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864072448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573508.101404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573508.101407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573508.101409:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573508.101413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573508.101424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573508.101432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573508/real 1788573508] req@ffff92fc67049500 x1875449864072576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573508.101442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864072576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573508.101447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573508.101449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573508.101452:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573508.549102:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573508.612810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573509.572846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573509.636810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573510.597055:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573510.660888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573511.620865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573511.684853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573512.644911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573512.708849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573513.220852:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573513.220865:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573513.220873:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573513.220887:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5263 00000100:00080000:3.0:1788573513.220901:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573513.220910:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573513.220970:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864072704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573513.221026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573513.221029:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573513.221031:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573513.221035:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5263 00000100:00080000:3.0:1788573513.221038:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573513.221042:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573513.221053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573513.221064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864072832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573513.221072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573513.221074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573513.221076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573513.221078:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5263 00000100:00080000:3.0:1788573513.221081:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573513.221084:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573513.221087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573513.221093:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864072960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573513.221105:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573513.221108:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573513.221109:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573513.221112:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5263 00000100:00080000:3.0:1788573513.221114:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573513.221116:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573513.221119:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573513.221129:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864073088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573513.221138:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573513.221144:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788573513.221146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864072704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573513.221146:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573513.221165:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5263 00000100:00080000:3.0:1788573513.221168:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573513.221171:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573513.221174:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573513.221191:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864073216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573513.221235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573513.221276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff92fc67339c00 x1875449864072704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573513.221290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864072704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573513.221298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573513.221302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573513.221305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864072704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573513.221310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573513.221323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573513.221331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff92fc67339880 x1875449864072832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573513.221339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864072832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573513.221344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573513.221346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573513.221348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573513.221352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573513.221363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573513.221370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff92fc67339180 x1875449864072960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573513.221378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864072960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573513.221383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573513.221385:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573513.221387:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864072960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573513.221409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573513.221421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573513.221427:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff92fc67339f80 x1875449864073088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573513.221435:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864073088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573513.221440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573513.221448:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573513.221450:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573513.221454:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573513.221465:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573513.221471:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573513/real 1788573513] req@ffff92fc6733ad80 x1875449864073216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573513.221479:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864073216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573513.221484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573513.221487:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573513.221489:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573513.668880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573513.732814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573514.692897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573514.756843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573515.716806:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573515.780816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573516.740892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573516.804819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573517.764803:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573517.828808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573518.340890:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573518.340898:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573518.340902:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573518.340911:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5268 00000100:00080000:1.0:1788573518.340919:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573518.340925:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573518.340958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864073344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573518.340980:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573518.341024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573518.341027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573518.341032:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5268 00000100:00100000:2.0:1788573518.341038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864073344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573518.341040:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573518.341048:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573518.341057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573518.341071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864073472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573518.341080:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573518.341083:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573518.341085:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573518.341087:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5268 00000100:00080000:1.0:1788573518.341092:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573518.341095:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573518.341098:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573518.341102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573518.341107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864073600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573518.341115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573518.341119:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573518.341120:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573518.341123:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5268 00000100:00080000:1.0:1788573518.341125:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573518.341127:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573518.341132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573518.341147:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864073728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573518.341147:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff92fc6728b100 x1875449864073344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573518.341172:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573518.341175:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573518.341177:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573518.341179:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5268 00000100:00080000:1.0:1788573518.341181:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573518.341183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864073344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573518.341186:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573518.341192:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573518.341198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788573518.341203:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864073856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573518.341206:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573518.341214:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864073344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573518.341240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573518.341256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573518.341265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff92fc67288380 x1875449864073472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573518.341273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864073472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573518.341279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573518.341281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573518.341284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573518.341296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573518.341309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573518.341316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff92fc67289c00 x1875449864073600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573518.341324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864073600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573518.341329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573518.341332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573518.341334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573518.341337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573518.341348:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573518.341354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff92fc6728a680 x1875449864073728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573518.341361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864073728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573518.341367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573518.341369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573518.341371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573518.341375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573518.341385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573518.341390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573518/real 1788573518] req@ffff92fc6728b800 x1875449864073856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573518.341398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864073856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573518.341403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573518.341405:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573518.341406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864073856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573518.788942:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573518.852871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573519.812997:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573519.876847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573520.836935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573520.900844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573521.860871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573521.924800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573522.884866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573522.948851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573523.460864:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573523.460875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573523.460879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573523.460895:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5274 00000100:00080000:3.0:1788573523.460907:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573523.460916:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573523.460951:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864073984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573523.461009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573523.461014:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573523.461016:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573523.461022:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5274 00000100:00080000:3.0:1788573523.461027:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573523.461032:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573523.461039:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864073984:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573523.461043:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573523.461062:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864074112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573523.461072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573523.461074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573523.461076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573523.461078:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5274 00000100:00080000:3.0:1788573523.461082:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573523.461085:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573523.461089:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573523.461098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864074240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573523.461104:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573523.461108:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573523.461109:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573523.461112:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5274 00000100:00080000:3.0:1788573523.461114:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573523.461116:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573523.461120:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573523.461124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573523.461131:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864074368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573523.461139:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573523.461141:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573523.461143:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573523.461145:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5274 00000100:00080000:3.0:1788573523.461148:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573523.461150:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573523.461153:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573523.461160:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864074496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573523.461178:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff92fc6733b100 x1875449864073984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573523.461199:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864073984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573523.461211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573523.461215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573523.461219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864073984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573523.461243:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573523.461262:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573523.461272:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff92fc67338380 x1875449864074112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573523.461280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864074112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573523.461290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573523.461292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573523.461295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573523.461299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573523.461310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573523.461316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff92fc67338a80 x1875449864074240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573523.461323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864074240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573523.461328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573523.461330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573523.461332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573523.461335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573523.461346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573523.461352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff92fc6733bb80 x1875449864074368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573523.461359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864074368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573523.461364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573523.461365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573523.461368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573523.461372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573523.461381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573523.461387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573523/real 1788573523] req@ffff92fc6733a300 x1875449864074496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573523.461394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864074496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573523.461399:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573523.461400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573523.461402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573523.908906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573523.972869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573524.932948:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573524.996885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573525.956884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573526.020777:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573526.980918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573527.044823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573528.004887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573528.068810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573528.580808:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573528.580819:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573528.580825:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573528.580837:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5279 00000100:00080000:3.0:1788573528.580848:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573528.580856:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573528.580891:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864074624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573528.580923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573528.580928:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573528.580931:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573528.580937:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5279 00000100:00080000:3.0:1788573528.580942:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573528.580948:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573528.580957:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864074624:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573528.580958:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573528.580979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864074752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573528.581009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573528.581014:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573528.581016:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573528.581022:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5279 00000100:00080000:3.0:1788573528.581028:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573528.581033:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573528.581039:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573528.581054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864074880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573528.581065:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573528.581069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573528.581072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573528.581078:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5279 00000100:00100000:1.0:1788573528.581084:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074752:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573528.581084:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573528.581094:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573528.581103:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573528.581121:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864075008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573528.581130:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074880:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573528.581140:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573528.581145:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573528.581147:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573528.581154:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5279 00000100:00080000:3.0:1788573528.581160:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573528.581161:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075008:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788573528.581169:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573528.581176:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573528.582841:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864075136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573528.582911:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573529.028959:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573529.092856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573530.052879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573530.116783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573531.076919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573531.140834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573531.653026:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573531.653064:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573531.653086:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573531.653116:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573531.653122:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573531.653128:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573531.653136:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573531.653175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff92fc67338700 x1875449864075136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573531.653203:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864075136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573531.653216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573531.653222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573531.653227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075136:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573531.653236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff92fc6733b800 x1875449864075008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573531.653250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864075008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573531.653260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573531.653264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573531.653268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573531.653275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff92fc67339500 x1875449864074752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573531.653290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864074752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573531.653300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573531.653304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573531.653308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573531.653315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff92fc6733b480 x1875449864074880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573531.653327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864074880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573531.653352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573531.653357:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573531.653362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864074880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573531.653370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573531] req@ffff92fc67338000 x1875449864074624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573531.653384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864074624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573531.653393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573531.653397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573531.653400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864074624:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573532.100912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573532.164831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573533.124917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573533.188824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573533.700818:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573533.700827:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573533.700831:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573533.700842:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5284 00000100:00080000:2.0:1788573533.700852:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573533.700860:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573533.700896:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864075264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573533.700919:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573533.700922:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573533.700924:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573533.700926:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5284 00000100:00080000:2.0:1788573533.700929:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573533.700933:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573533.700941:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573533.700950:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864075392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573533.700958:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573533.700961:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573533.700962:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573533.700964:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5284 00000100:00080000:2.0:1788573533.700967:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573533.700969:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573533.700972:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573533.700980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864075264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573533.700981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864075520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573533.701017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573533.701022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573533.701024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573533.701029:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5284 00000100:00080000:2.0:1788573533.701032:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573533.701042:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573533.701048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573533.701074:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864075648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573533.701085:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573533.701087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573533.701093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573533.701095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573533.701100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5284 00000100:00080000:2.0:1788573533.701106:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573533.701111:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573533.701117:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573533.701132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864075776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573533.701140:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff92fc4617c000 x1875449864075264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573533.701173:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864075264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573533.701186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573533.701191:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573533.701194:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864075264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573533.701203:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573533.701229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573533.701245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff92fc4617fb80 x1875449864075392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573533.701260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864075392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573533.701269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573533.701273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573533.701277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573533.701306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573533.701329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573533.701341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff92fc4617c380 x1875449864075520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573533.701354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864075520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573533.701362:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573533.701365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573533.701368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573533.701374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573533.701393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573533.701404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff92fc4617d180 x1875449864075648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573533.701416:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864075648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573533.701424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573533.701427:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573533.701430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573533.701436:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573533.701455:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573533.701467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff92fc4617e680 x1875449864075776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573533.701478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864075776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573533.701486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573533.701489:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573533.701493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864075776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573534.148910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573534.212851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573535.172845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573535.236820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573536.196892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573536.260877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573537.220874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573537.284832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573538.244866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573538.308854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573538.820815:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.820825:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573538.820829:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.820838:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5289 00000100:00080000:1.0:1788573538.820847:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.820853:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573538.820886:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864075904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573538.820921:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.820924:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573538.820926:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.820929:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5289 00000100:00080000:1.0:1788573538.820931:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.820934:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573538.820943:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573538.820951:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864075904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573538.820953:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864076032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573538.820974:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.820976:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573538.820978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.820980:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5289 00000100:00080000:1.0:1788573538.820983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.820985:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573538.820987:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573538.821010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864076160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573538.821018:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.821020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573538.821022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.821024:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5289 00000100:00080000:1.0:1788573538.821027:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.821030:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573538.821033:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573538.821043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864076288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573538.821051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573538.821051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573538.821057:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573538.821058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.821061:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5289 00000100:00080000:1.0:1788573538.821064:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.821066:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573538.821070:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573538.821081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864076416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573538.821097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff92fc67288e00 x1875449864075904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.821113:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864075904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.821121:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.821124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573538.821127:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864075904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573538.821146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573538.821163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573538.821172:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff92fc67289f80 x1875449864076032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.821180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864076032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.821186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.821188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573538.821191:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573538.821194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573538.821205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573538.821211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff92fc67288000 x1875449864076160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.821218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864076160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.821223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.821225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573538.821227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573538.821231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573538.821241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573538.821246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff92fc67289880 x1875449864076288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.821253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864076288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.821258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.821260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573538.821262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573538.821266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573538.821275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573538.821281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff92fc6728aa00 x1875449864076416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.821288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864076416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.821293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.821295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573538.821297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573539.268925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573539.332869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573540.292873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573540.356911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573541.316861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573541.380825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573542.340851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573542.404837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573543.364946:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573543.428830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573543.940857:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.940866:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573543.940870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.940897:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5294 00000100:00080000:1.0:1788573543.940906:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.940913:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573543.940991:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864076544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573543.941040:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.941049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573543.941057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.941061:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5294 00000100:00080000:1.0:1788573543.941064:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.941068:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573543.941077:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573543.941089:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864076544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573543.941137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864076672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573543.941147:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.941150:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573543.941152:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.941154:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5294 00000100:00080000:1.0:1788573543.941157:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.941160:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788573543.941166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788573543.941175:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573543.941202:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864076800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573543.941215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff92fc6728b480 x1875449864076544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573543.941216:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.941235:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573543.941238:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573543.941240:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864076544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573543.941253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573543.941258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573543.941262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864076544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573543.941286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076672:192.168.202.132@tcp:38:kworker.0 00000100:00080000:1.0:1788573543.941295:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5294 00000100:00080000:1.0:1788573543.941299:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.941302:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788573543.941313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788573543.941316:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573543.941331:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864076928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573543.941331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff92fc6728ad80 x1875449864076672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573543.941349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864076672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573543.941358:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.941366:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573543.941369:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573543.941371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573543.941376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573543.941382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076672:192.168.202.132@tcp:38:kworker.0 00000100:00080000:1.0:1788573543.941384:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5294 00000100:00080000:1.0:1788573543.941390:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.941397:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573543.941398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076800:192.168.202.132@tcp:38:kworker.0 00000008:00100000:1.0:1788573543.941410:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573543.941425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573543.941433:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864077056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573543.941441:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff92fc6728a300 x1875449864076800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573543.941457:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864076800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573543.941466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573543.941470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573543.941474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573543.941490:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573543.941509:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573543.941519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff92fc6728b480 x1875449864076928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573543.941530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864076928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573543.941548:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573543.941552:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573543.941555:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864076928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573543.941561:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573543.941580:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573543.941591:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573543] req@ffff92fc6728aa00 x1875449864077056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573543.941603:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864077056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573543.941611:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573543.941614:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573543.941617:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573544.388869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573544.452814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573545.412912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573545.476846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573546.436887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573546.500792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573547.460933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573547.524779:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573548.484855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573548.548784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573549.060796:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573549.060807:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573549.060812:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573549.060822:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5299 00000100:00080000:1.0:1788573549.060834:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573549.060841:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573549.060885:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864077184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573549.060913:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573549.060917:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573549.060918:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573549.060921:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5299 00000100:00080000:1.0:1788573549.060924:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573549.060926:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573549.060936:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573549.060946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864077312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573549.060953:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573549.060957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573549.060959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573549.060962:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5299 00000100:00080000:1.0:1788573549.060965:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573549.060968:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573549.060971:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573549.060977:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864077440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573549.060990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573549.060993:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573549.060995:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573549.061003:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5299 00000100:00100000:2.0:1788573549.061003:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864077184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573549.061009:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573549.061030:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573549.061035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573549.061048:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864077568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573549.061058:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573549.061062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573549.061064:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573549.061070:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5299 00000100:00080000:1.0:1788573549.061075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573549.061081:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573549.061087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573549.061106:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864077696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573549.061118:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573549.061171:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573549/real 1788573549] req@ffff92fc67289880 x1875449864077184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573549.061193:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864077184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573549.061206:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573549.061211:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573549.061216:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864077184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573549.061223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573549.061244:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573549.061254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573549/real 1788573549] req@ffff92fc67288000 x1875449864077312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573549.061263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864077312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573549.061268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573549.061271:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573549.061273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573549.061294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573549.061306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573549.061313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573549/real 1788573549] req@ffff92fc67289f80 x1875449864077440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573549.061320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864077440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573549.061326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573549.061328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573549.061330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573549.061334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573549.061344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573549.061350:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573549/real 1788573549] req@ffff92fc67288e00 x1875449864077568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573549.061359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864077568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573549.061367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573549.061370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573549.061374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573549.061380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573549.061401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573549.061413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573549/real 1788573549] req@ffff92fc6728b800 x1875449864077696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573549.061425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864077696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573549.061430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573549.061433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573549.061435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573549.508915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573549.572790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573550.532908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573550.596852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573551.556963:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573551.620862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573552.580844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573552.644837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573553.604947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573553.668864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573554.180867:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573554.180879:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573554.180884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573554.180895:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5304 00000100:00080000:2.0:1788573554.180907:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573554.180914:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573554.180967:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864077824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573554.180993:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573554.180996:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573554.180997:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573554.181000:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5304 00000100:00080000:2.0:1788573554.181003:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573554.181007:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573554.181029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573554.181044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864077952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573554.181054:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864077824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573554.181055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573554.181068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573554.181069:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573554.181073:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5304 00000100:00080000:2.0:1788573554.181076:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573554.181079:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573554.181084:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573554.181095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864078080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573554.181106:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573554.181110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573554.181112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573554.181116:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5304 00000100:00080000:2.0:1788573554.181121:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573554.181126:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573554.181133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788573554.181134:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573554.181153:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864078208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573554.181164:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573554.181167:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573554.181169:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573554.181175:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5304 00000100:00080000:2.0:1788573554.181179:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573554.181184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff92fc4617dc00 x1875449864077824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573554.181189:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573554.181201:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573554.181211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864077824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573554.181223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573554.181229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573554.181229:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864078336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573554.181235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864077824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573554.181270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573554.181287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573554.181297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff92fc4617ca80 x1875449864077952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573554.181306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864077952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573554.181313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573554.181316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573554.181318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864077952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573554.181322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573554.181334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573554.181340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff92fc4617c700 x1875449864078080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573554.181354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864078080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573554.181359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573554.181361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573554.181363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573554.181367:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573554.181390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573554.181398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff92fc4617f800 x1875449864078208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573554.181406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864078208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573554.181412:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573554.181415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573554.181417:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573554.181421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573554.181431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573554.181437:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573554/real 1788573554] req@ffff92fc4617f100 x1875449864078336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573554.181448:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864078336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573554.181459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573554.181461:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573554.181463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573554.628865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573554.693120:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573555.652926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573555.716843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573556.676824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573556.740823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573557.700927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573557.764832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573558.724887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573558.788824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573559.300855:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573559.300865:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573559.300869:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573559.300881:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5309 00000100:00080000:0.0:1788573559.300892:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573559.300900:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573559.300939:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864078464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573559.300968:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573559.300972:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573559.300975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573559.300979:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5309 00000100:00080000:0.0:1788573559.300983:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573559.300988:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573559.300998:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573559.301012:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864078464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573559.301015:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864078592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573559.301046:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573559.301051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573559.301053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573559.301058:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5309 00000100:00080000:0.0:1788573559.301082:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573559.301086:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573559.301092:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573559.301094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573559.301108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864078720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573559.301120:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573559.301124:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573559.301126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573559.301131:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5309 00000100:00080000:0.0:1788573559.301135:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573559.301140:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573559.301145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff92fc6704b100 x1875449864078464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573559.301148:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573559.301165:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864078848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573559.301174:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864078464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573559.301181:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573559.301187:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573559.301187:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573559.301194:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573559.301194:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573559.301201:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864078464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573559.301204:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5309 00000100:00080000:0.0:1788573559.301211:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573559.301215:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573559.301221:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573559.301224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573559.301238:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864078976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573559.301248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573559.301262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff92fc6704ad80 x1875449864078592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573559.301271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864078592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573559.301278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573559.301282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573559.301284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573559.301288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573559.301309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573559.301321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff92fc6704a680 x1875449864078720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573559.301329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864078720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573559.301339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573559.301343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573559.301346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573559.301352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573559.301372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573559.301384:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff92fc6704aa00 x1875449864078848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573559.301394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864078848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573559.301399:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573559.301401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573559.301404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573559.301415:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573559.301434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573559.301442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573559/real 1788573559] req@ffff92fc6704a300 x1875449864078976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573559.301450:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864078976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573559.301455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573559.301457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573559.301460:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864078976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573559.748906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573559.812844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573560.772824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573560.836817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573561.796897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573561.860838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573562.820936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573562.884808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573563.844937:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573563.908864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573564.420830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573564.420839:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573564.420843:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573564.420852:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5315 00000100:00080000:0.0:1788573564.420860:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573564.420868:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573564.420898:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864079104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573564.420921:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573564.420924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573564.420926:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573564.420929:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5315 00000100:00080000:0.0:1788573564.420931:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573564.420934:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573564.420942:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573564.420955:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864079232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573564.420965:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864079104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573564.420966:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573564.420975:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573564.420979:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573564.420986:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5315 00000100:00080000:0.0:1788573564.420991:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573564.420995:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573564.421001:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573564.421018:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864079360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573564.421030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573564.421048:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573564.421049:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573564.421061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573564.421064:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5315 00000100:00080000:0.0:1788573564.421072:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573564.421078:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573564.421085:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573564.421097:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864079488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573564.421104:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573564.421107:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573564.421111:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573564.421112:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff92fc67049880 x1875449864079104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573564.421115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5315 00000100:00080000:0.0:1788573564.421120:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573564.421124:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573564.421131:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573564.421142:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864079616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573564.421147:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864079104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573564.421160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573564.421164:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573564.421168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864079104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573564.421190:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573564.421210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573564.421223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff92fc67048000 x1875449864079232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573564.421234:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864079232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573564.421240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573564.421243:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573564.421245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573564.421249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573564.421262:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573564.421269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff92fc67049f80 x1875449864079360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573564.421276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864079360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573564.421282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573564.421285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573564.421288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573564.421292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573564.421304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573564.421310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff92fc67048700 x1875449864079488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573564.421318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864079488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573564.421323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573564.421325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573564.421327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573564.421330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573564.421340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573564.421346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573564/real 1788573564] req@ffff92fc67048380 x1875449864079616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573564.421353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864079616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573564.421358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573564.421360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573564.421362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573564.868881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573564.932817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573565.892886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573565.956817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573566.916865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573566.980808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573567.940896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573568.004863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573568.964872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573569.028864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573569.540860:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573569.540870:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573569.540875:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573569.540885:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5320 00000100:00080000:1.0:1788573569.540896:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573569.540903:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573569.540940:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864079744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573569.540968:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573569.540972:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573569.540974:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573569.540979:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5320 00000100:00080000:1.0:1788573569.540991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573569.540996:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573569.541007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573569.541015:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864079744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573569.541020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864079872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573569.541031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573569.541034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573569.541035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573569.541050:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5320 00000100:00080000:1.0:1788573569.541053:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573569.541056:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573569.541060:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573569.541070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864080000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573569.541078:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573569.541082:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573569.541084:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573569.541087:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5320 00000100:00080000:1.0:1788573569.541092:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573569.541095:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573569.541101:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573569.541115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864080128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573569.541118:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573569.541131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573569.541135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573569.541137:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573569.541141:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5320 00000100:00080000:1.0:1788573569.541144:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573569.541147:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573569.541151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573569.541162:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864080256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573569.541171:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff92fc6728a680 x1875449864079744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573569.541194:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864079744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573569.541213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573569.541217:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573569.541221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864079744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573569.541247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573569.541273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573569.541286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff92fc67289c00 x1875449864079872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573569.541299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864079872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573569.541308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573569.541311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573569.541315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864079872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573569.541322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573569.541341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573569.541353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff92fc67288380 x1875449864080000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573569.541366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864080000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573569.541375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573569.541378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573569.541382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573569.541389:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573569.541411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573569.541422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff92fc6728b100 x1875449864080128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573569.541435:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864080128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573569.541455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573569.541458:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573569.541462:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573569.541468:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573569.541490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573569.541502:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573569/real 1788573569] req@ffff92fc67288a80 x1875449864080256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573569.541514:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864080256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573569.541523:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573569.541527:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573569.541530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573569.988927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573570.052836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573571.012925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573571.076836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573572.036853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573572.100834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573573.060880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573573.124801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573574.084908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573574.148801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573574.660853:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.660862:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573574.660866:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.660874:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5325 00000100:00080000:0.0:1788573574.660883:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.660889:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573574.660923:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864080384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.660950:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.660955:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.660957:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.660961:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5325 00000100:00080000:0.0:1788573574.660966:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.660971:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573574.660981:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573574.660997:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864080384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573574.660999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864080512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.661016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.661020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573574.661022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.661027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5325 00000100:00080000:0.0:1788573574.661031:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.661035:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573574.661040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573574.661066:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864080640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.661077:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.661080:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.661085:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573574.661085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573574.661093:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5325 00000100:00080000:0.0:1788573574.661098:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.661103:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573574.661110:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573574.661133:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864080768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573574.661135:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff92fc67048a80 x1875449864080384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573574.661170:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573574.661173:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864080384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573574.661175:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573574.661179:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.661189:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5325 00000100:00080000:2.0:1788573574.661190:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573574.661193:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573574.661196:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788573574.661201:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573574.661204:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864080384:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788573574.661208:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573574.661220:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864080896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573574.661227:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573574.661251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573574.661261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff92fc67048380 x1875449864080512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573574.661270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864080512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573574.661276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573574.661278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573574.661281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573574.661284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573574.661296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573574.661302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff92fc67048700 x1875449864080640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573574.661310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864080640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573574.661320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573574.661322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573574.661324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573574.661327:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573574.661338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573574.661344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff92fc67049f80 x1875449864080768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573574.661351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864080768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573574.661365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573574.661367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573574.661370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573574.661382:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573574.661394:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573574.661401:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff92fc67048000 x1875449864080896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573574.661409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864080896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573574.661414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573574.661416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573574.661419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864080896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573575.108896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573575.172828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573576.132909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573576.196811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573577.156875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573577.220845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573578.180871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573578.244823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573579.204955:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573579.268870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573579.780854:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573579.780868:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573579.780873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573579.780884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5330 00000100:00080000:1.0:1788573579.780897:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573579.780907:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573579.780960:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864081024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573579.780997:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573579.781002:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573579.781005:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573579.781010:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5330 00000100:00080000:1.0:1788573579.781016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573579.781021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573579.781036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573579.781044:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864081024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573579.781069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864081152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573579.781084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573579.781088:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573579.781091:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573579.781096:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5330 00000100:00080000:1.0:1788573579.781101:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573579.781106:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573579.781113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573579.781140:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573579.781141:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864081280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573579.781155:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573579.781159:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573579.781162:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573579.781167:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5330 00000100:00080000:1.0:1788573579.781172:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573579.781177:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573579.781183:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573579.781192:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff92fc67288700 x1875449864081024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573579.781211:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864081408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573579.781219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864081024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573579.781233:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573579.781239:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573579.781240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573579.781243:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573579.781247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:1.0:1788573579.781253:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5330 00000100:00100000:2.0:1788573579.781255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864081024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573579.781262:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573579.781270:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573579.781277:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573579.781294:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864081536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573579.781295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573579.781325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573579.781339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff92fc6728bb80 x1875449864081152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573579.781353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864081152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573579.781363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573579.781367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573579.781370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573579.781376:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573579.781393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573579.781402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff92fc67289500 x1875449864081280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573579.781424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864081280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573579.781437:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573579.781440:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573579.781444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573579.781449:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573579.781467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573579.781476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff92fc67289180 x1875449864081408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573579.781489:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864081408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573579.781497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573579.781501:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573579.781504:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573579.781518:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573579.781533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573579.781542:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff92fc67288700 x1875449864081536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573579.781554:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864081536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573579.781563:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573579.781566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573579.781569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573580.228993:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573580.292850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573581.252894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573581.316812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573582.276887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573582.340834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573583.300820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573583.364839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573584.324877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573584.388789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573584.900840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573584.900851:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573584.900855:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573584.900867:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5335 00000100:00080000:1.0:1788573584.900877:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573584.900885:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573584.900922:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864081664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573584.900953:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573584.900958:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573584.900961:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573584.900965:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5335 00000100:00080000:1.0:1788573584.900970:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573584.900974:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573584.900985:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573584.901002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864081792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573584.901021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573584.901025:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573584.901027:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573584.901032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5335 00000100:00080000:1.0:1788573584.901036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573584.901044:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573584.901046:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864081664:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788573584.901051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573584.901087:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864081920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573584.901102:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573584.901106:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573584.901108:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573584.901113:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5335 00000100:00080000:1.0:1788573584.901118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573584.901124:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573584.901130:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573584.901144:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864082048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573584.901156:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573584.901166:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788573584.901166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573584.901175:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573584.901181:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5335 00000100:00080000:1.0:1788573584.901186:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573584.901192:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573584.901198:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573584.901212:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864082176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573584.901239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff92fc67288a80 x1875449864081664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573584.901265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864081664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573584.901278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573584.901283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573584.901287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864081664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573584.901296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573584.901323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573584.901339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff92fc6728b100 x1875449864081792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573584.901353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864081792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573584.901362:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573584.901365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573584.901369:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573584.901408:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573584.901430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573584.901442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff92fc67288380 x1875449864081920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573584.901455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864081920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573584.901464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573584.901468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573584.901471:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864081920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573584.901478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573584.901499:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573584.901511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff92fc67289c00 x1875449864082048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573584.901523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864082048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573584.901532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573584.901536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573584.901540:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573584.901547:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573584.901566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573584.901577:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff92fc6728a680 x1875449864082176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573584.901591:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864082176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573584.901599:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573584.901603:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573584.901606:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573585.348866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573585.412859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573586.372917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573586.436816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573587.396874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573587.460824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573588.420872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573588.484814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573589.444861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573589.508855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573590.020877:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573590.020888:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573590.020893:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573590.020905:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5340 00000100:00080000:3.0:1788573590.020916:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573590.020924:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573590.020963:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864082304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573590.020999:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573590.021026:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573590.021028:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573590.021034:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5340 00000100:00080000:3.0:1788573590.021040:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573590.021045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573590.021049:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864082304:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573590.021057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573590.021078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864082432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573590.021094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573590.021098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573590.021101:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573590.021105:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5340 00000100:00080000:3.0:1788573590.021109:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573590.021114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573590.021118:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573590.021126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573590.021140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864082560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573590.021150:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573590.021155:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788573590.021156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573590/real 1788573590] req@ffff92fc6733a680 x1875449864082304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573590.021159:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573590.021171:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5340 00000100:00080000:3.0:1788573590.021177:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573590.021183:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573590.021187:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864082304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573590.021189:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573590.021203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573590.021209:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573590.021209:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864082688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573590.021215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864082304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573590.021232:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573590.021235:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573590.021238:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573590.021241:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082432:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573590.021246:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5340 00000100:00080000:3.0:1788573590.021253:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573590.021258:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788573590.021264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573590.021264:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573590.021275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573590/real 1788573590] req@ffff92fc67338e00 x1875449864082432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573590.021283:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864082816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573590.021294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864082432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573590.021314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573590.021319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573590.021325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573590.021332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573590.021350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573590.021359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573590/real 1788573590] req@ffff92fc6733aa00 x1875449864082560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573590.021367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864082560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573590.021373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573590.021375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573590.021377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573590.021380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573590.021392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573590.021398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573590/real 1788573590] req@ffff92fc67338000 x1875449864082688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573590.021405:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864082688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573590.021410:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573590.021412:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573590.021415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573590.021426:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573590.021437:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573590.021444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573590/real 1788573590] req@ffff92fc6733b480 x1875449864082816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573590.021451:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864082816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573590.021456:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573590.021457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573590.021459:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864082816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573590.468933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573590.532846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573591.492847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573591.556846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573592.516927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573592.580854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573593.540916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573593.604830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573594.564924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573594.628828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573595.140781:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573595.140788:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573595.140792:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573595.140801:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5345 00000100:00080000:3.0:1788573595.140810:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573595.140816:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573595.140855:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864082944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573595.140875:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573595.140879:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573595.140880:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573595.140882:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5345 00000100:00080000:3.0:1788573595.140885:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573595.140888:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573595.140902:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573595.140912:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864083072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573595.140919:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573595.140922:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573595.140923:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573595.140926:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5345 00000100:00080000:3.0:1788573595.140929:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573595.140945:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864082944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573595.140949:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573595.140957:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573595.140970:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864083200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573595.140982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573595.140986:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573595.140988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573595.140992:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5345 00000100:00080000:3.0:1788573595.141007:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573595.141012:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573595.141019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573595.141031:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864083328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573595.141048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573595.141052:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573595.141055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573595.141058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083072:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573595.141063:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5345 00000100:00080000:3.0:1788573595.141071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573595.141075:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573595.141080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573595.141092:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864083456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573595.141100:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573595.141119:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573595.141146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573595.588863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573595.652829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573596.612984:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573596.676889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573597.636852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573597.700817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573598.213112:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573598.213175:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573598.213202:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573598.213247:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573598.213259:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573598.213268:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573598.213277:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788573598.213393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573595/real 1788573598] req@ffff92fc6733bb80 x1875449864083456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573598.213425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864083456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573598.213448:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573598.213453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573598.213458:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083456:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573598.213467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573595/real 1788573598] req@ffff92fc67338700 x1875449864083200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573598.213482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864083200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573598.213491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573598.213496:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573598.213500:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573598.213508:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573595/real 1788573598] req@ffff92fc6733a300 x1875449864083328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573598.213522:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864083328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573598.213531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573598.213535:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573598.213539:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573598.213548:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573595/real 1788573598] req@ffff92fc67339500 x1875449864082944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573598.213563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864082944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573598.213572:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573598.213576:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573598.213580:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864082944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573598.213587:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573595/real 1788573598] req@ffff92fc6733b800 x1875449864083072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573598.213600:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864083072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573598.213608:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573598.213611:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573598.213614:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083072:192.168.202.132@tcp:38:kworker.0 00010000:00010000:1.0:1788573598.660923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573598.724836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573599.684980:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573599.748862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573600.260877:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573600.260893:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573600.260899:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573600.260913:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5350 00000100:00080000:3.0:1788573600.260928:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573600.260941:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573600.260990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864083584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573600.261032:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573600.261037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573600.261040:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573600.261047:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5350 00000100:00080000:3.0:1788573600.261052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573600.261058:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573600.261072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573600.261096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864083712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573600.261098:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864083584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573600.261120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573600.261125:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573600.261128:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573600.261133:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5350 00000100:00080000:3.0:1788573600.261138:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573600.261143:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573600.261149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573600.261164:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864083840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573600.261190:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573600.261191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573600.261199:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573600.261202:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573600.261208:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5350 00000100:00080000:3.0:1788573600.261214:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573600.261219:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573600.261226:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573600.261240:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864083968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573600.261252:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573600.261257:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573600.261260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff92fc67338a80 x1875449864083584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573600.261261:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573600.261270:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5350 00000100:00080000:3.0:1788573600.261278:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573600.261285:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573600.261292:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573600.261299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864083584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573600.261307:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864084096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573600.261316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573600.261324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573600.261330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864083584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573600.261374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573600.261402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573600.261418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff92fc67338380 x1875449864083712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573600.261431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864083712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573600.261439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573600.261442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573600.261446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573600.261452:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573600.261473:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573600.261490:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff92fc6733b100 x1875449864083840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573600.261504:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864083840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573600.261512:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573600.261515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573600.261519:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573600.261525:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573600.261548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573600.261559:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff92fc6733ad80 x1875449864083968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573600.261587:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864083968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573600.261596:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573600.261599:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573600.261603:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864083968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573600.261610:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573600.261633:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573600.261647:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573600/real 1788573600] req@ffff92fc67339f80 x1875449864084096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573600.261659:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864084096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573600.261668:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573600.261671:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573600.261675:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573600.709022:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573600.772908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573601.732941:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573601.796816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573602.756892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573602.820851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573603.780870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573603.844794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573604.804933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573604.868881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573605.380899:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573605.380910:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573605.380916:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573605.380928:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5355 00000100:00080000:1.0:1788573605.380940:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573605.380948:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573605.380994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864084224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573605.381023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573605.381026:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573605.381027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573605.381030:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5355 00000100:00080000:1.0:1788573605.381032:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573605.381035:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573605.381044:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573605.381054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864084352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573605.381062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573605.381065:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573605.381066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573605.381069:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5355 00000100:00080000:1.0:1788573605.381070:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573605.381073:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573605.381079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573605.381079:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864084224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573605.381088:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864084480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573605.381099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573605.381101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573605.381101:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573605.381104:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5355 00000100:00080000:1.0:1788573605.381106:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573605.381108:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573605.381112:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573605.381119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864084608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573605.381124:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573605.381126:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573605.381127:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573605.381130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5355 00000100:00080000:1.0:1788573605.381132:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573605.381135:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573605.381139:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573605.381145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864084736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573605.381168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573605.381225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff92fc6728b800 x1875449864084224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573605.381250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864084224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573605.381264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573605.381268:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573605.381273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864084224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573605.381283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573605.381315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573605.381331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff92fc67288e00 x1875449864084352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573605.381346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864084352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573605.381356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573605.381360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573605.381363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573605.381396:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573605.381419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573605.381431:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff92fc67289f80 x1875449864084480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573605.381444:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864084480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573605.381454:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573605.381456:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573605.381459:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573605.381463:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573605.381479:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573605.381487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff92fc67288000 x1875449864084608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573605.381500:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864084608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573605.381505:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573605.381507:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573605.381509:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573605.381512:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573605.381523:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573605.381529:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573605/real 1788573605] req@ffff92fc67289880 x1875449864084736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573605.381536:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864084736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573605.381541:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573605.381543:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573605.381545:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573605.828834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573605.892822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573606.852896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573606.916844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573607.876931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573607.940865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573608.900972:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573608.964886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573609.924877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573609.988810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573610.500855:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573610.500865:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573610.500870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573610.500881:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5361 00000100:00080000:3.0:1788573610.500891:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573610.500900:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573610.500933:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864084864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573610.500956:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573610.500959:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573610.500961:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573610.500964:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5361 00000100:00080000:3.0:1788573610.500967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573610.500969:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573610.500977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573610.500994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864084992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573610.501007:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573610.501009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573610.501011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573610.501014:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5361 00000100:00080000:3.0:1788573610.501016:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573610.501021:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864084864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573610.501022:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573610.501029:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573610.501039:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864085120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573610.501050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573610.501054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573610.501056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573610.501060:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5361 00000100:00080000:3.0:1788573610.501080:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573610.501085:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788573610.501091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573610.501092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573610.501111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864085248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573610.501123:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573610.501126:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573610.501129:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573610.501133:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5361 00000100:00080000:3.0:1788573610.501138:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573610.501142:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573610.501147:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff92fc67339180 x1875449864084864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573610.501149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573610.501170:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864085376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573610.501177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864084864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573610.501201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573610.501206:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573610.501211:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864084864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573610.501218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573610.501241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573610.501254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff92fc67339880 x1875449864084992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573610.501267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864084992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573610.501275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573610.501279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573610.501282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864084992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573610.501308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573610.501328:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573610.501340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff92fc67339c00 x1875449864085120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573610.501354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864085120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573610.501374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573610.501378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573610.501382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573610.501390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573610.501413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573610.501426:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff92fc67339f80 x1875449864085248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573610.501439:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864085248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573610.501448:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573610.501452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573610.501465:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573610.501473:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573610.501495:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573610.501508:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573610/real 1788573610] req@ffff92fc6733ad80 x1875449864085376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573610.501521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864085376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573610.501529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573610.501533:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573610.501537:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573610.948883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573611.012791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573611.972907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573612.036798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573612.996875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573613.060848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573614.020968:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573614.084840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573615.044930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573615.108837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573615.620841:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573615.620851:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573615.620855:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573615.620866:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5366 00000100:00080000:3.0:1788573615.620877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573615.620886:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573615.620924:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864085504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573615.620953:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573615.620957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573615.620959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573615.620963:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5366 00000100:00080000:3.0:1788573615.620968:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573615.620972:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573615.620982:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573615.620997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864085632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573615.621001:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864085504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573615.621016:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573615.621021:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573615.621022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573615.621026:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5366 00000100:00080000:3.0:1788573615.621029:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573615.621033:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573615.621046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573615.621055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864085760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573615.621062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573615.621066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573615.621067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573615.621070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573615.621074:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5366 00000100:00080000:3.0:1788573615.621078:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573615.621080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573615.621084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573615.621092:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864085888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573615.621098:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573615.621101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573615.621103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573615.621105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5366 00000100:00080000:3.0:1788573615.621107:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573615.621109:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573615.621114:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff92fc6733b100 x1875449864085504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573615.621115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573615.621127:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864086016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573615.621135:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864085504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573615.621144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573615.621148:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573615.621152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864085504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573615.621172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573615.621187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573615.621196:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff92fc67338380 x1875449864085632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573615.621205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864085632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573615.621211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573615.621213:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573615.621215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573615.621219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573615.621229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573615.621235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff92fc67338a80 x1875449864085760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573615.621242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864085760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573615.621253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573615.621255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573615.621257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573615.621260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573615.621270:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573615.621276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff92fc6733b800 x1875449864085888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573615.621283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864085888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573615.621288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573615.621297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573615.621300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864085888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573615.621303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573615.621315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573615.621321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff92fc67339500 x1875449864086016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573615.621328:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864086016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573615.621333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573615.621335:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573615.621338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573616.068865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573616.132780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573617.092931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573617.156808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573618.116843:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573618.180830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573619.140892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573619.204817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573620.164822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573620.228852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573620.740847:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573620.740855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573620.740858:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573620.740867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5371 00000100:00080000:1.0:1788573620.740877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573620.740882:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573620.740921:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864086144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573620.740945:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573620.740950:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573620.740952:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573620.740956:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5371 00000100:00080000:1.0:1788573620.740960:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573620.740965:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573620.740976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573620.740987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864086144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573620.740993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864086272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573620.741008:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573620.741012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573620.741014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573620.741018:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5371 00000100:00080000:1.0:1788573620.741021:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573620.741025:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573620.741030:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573620.741039:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864086400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573620.741047:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573620.741049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573620.741051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573620.741056:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5371 00000800:00000100:2.0:1788573620.741056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573620.741068:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573620.741071:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573620.741075:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573620.741084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864086528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573620.741091:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573620.741093:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573620.741096:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573620.741097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff92fc6728aa00 x1875449864086144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.741100:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5371 00000100:00080000:1.0:1788573620.741105:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573620.741111:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573620.741117:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573620.741123:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864086144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.741127:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864086656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573620.741145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573620.741150:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573620.741154:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864086144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573620.741173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573620.741188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573620.741197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff92fc6728b480 x1875449864086272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573620.741205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864086272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573620.741211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573620.741213:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573620.741216:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573620.741219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573620.741229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573620.741236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff92fc6728a300 x1875449864086400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573620.741243:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864086400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573620.741248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573620.741250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573620.741252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573620.741255:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573620.741265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573620.741271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff92fc6728ad80 x1875449864086528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573620.741278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864086528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573620.741283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573620.741284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573620.741286:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573620.741290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573620.741300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573620.741305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff92fc67289880 x1875449864086656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573620.741312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864086656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573620.741318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573620.741320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573620.741322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573621.188889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573621.252791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573622.212961:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573622.276864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573623.236987:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573623.300811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573624.260880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573624.324816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573625.284900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573625.348792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573625.860804:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573625.860815:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573625.860820:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573625.860831:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5376 00000100:00080000:1.0:1788573625.860843:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573625.860851:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573625.860897:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864086784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573625.860924:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573625.860928:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573625.860930:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573625.860932:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5376 00000100:00080000:1.0:1788573625.860935:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573625.860937:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573625.860947:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573625.860963:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864086912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573625.860971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573625.860974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573625.860976:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573625.860979:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5376 00000100:00080000:1.0:1788573625.860981:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573625.860983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573625.860988:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573625.860995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864087040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573625.861002:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573625.861005:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573625.861008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573625.861010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864086784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573625.861012:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5376 00000100:00080000:1.0:1788573625.861017:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573625.861020:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573625.861024:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573625.861032:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864087168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573625.861039:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573625.861041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573625.861043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573625.861045:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5376 00000100:00080000:1.0:1788573625.861047:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573625.861050:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573625.861053:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573625.861060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864087296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573625.861089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573625.861140:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff92fc67288000 x1875449864086784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573625.861157:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864086784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573625.861166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573625.861169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573625.861172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864086784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573625.861178:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573625.861196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573625.861205:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff92fc67289f80 x1875449864086912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573625.861213:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864086912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573625.861220:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573625.861222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573625.861224:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864086912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573625.861250:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573625.861262:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573625.861268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff92fc67288e00 x1875449864087040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573625.861276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864087040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573625.861281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573625.861283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573625.861285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573625.861288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573625.861300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573625.861306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff92fc6728b800 x1875449864087168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573625.861313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864087168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573625.861318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573625.861319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573625.861321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573625.861324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573625.861334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573625.861340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff92fc6728a680 x1875449864087296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573625.861347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864087296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573625.861353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573625.861354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573625.861356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573626.308899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573626.372886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573627.332910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573627.396803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573628.356838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573628.420858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573629.380899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573629.444839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573630.404896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573630.468858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573630.980865:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573630.980875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573630.980878:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573630.980888:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5381 00000100:00080000:3.0:1788573630.980897:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573630.980904:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573630.980948:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864087424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573630.980975:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573630.980978:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573630.980980:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573630.980983:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5381 00000100:00080000:3.0:1788573630.980986:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573630.980988:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573630.980996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573630.981008:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864087552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573630.981017:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573630.981020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573630.981021:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573630.981027:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5381 00000100:00100000:2.0:1788573630.981029:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864087424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573630.981030:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573630.981039:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573630.981044:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573630.981053:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864087680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573630.981060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573630.981064:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573630.981066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573630.981069:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5381 00000100:00080000:3.0:1788573630.981071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573630.981074:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573630.981078:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573630.981085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864087808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573630.981092:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573630.981095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573630.981096:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573630.981099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5381 00000800:00000100:2.0:1788573630.981105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573630.981105:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573630.981113:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573630.981117:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573630.981126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864087936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573630.981158:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff92fc6733a300 x1875449864087424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573630.981182:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864087424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573630.981196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573630.981200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573630.981205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864087424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573630.981232:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573630.981254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573630.981265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff92fc67338700 x1875449864087552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573630.981274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864087552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573630.981286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573630.981289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573630.981291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573630.981295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573630.981307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573630.981315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff92fc6733bb80 x1875449864087680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573630.981323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864087680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573630.981328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573630.981330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573630.981333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573630.981337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573630.981348:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573630.981372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff92fc6733b480 x1875449864087808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573630.981381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864087808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573630.981387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573630.981389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573630.981392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573630.981397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573630.981410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573630.981417:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff92fc67338000 x1875449864087936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573630.981475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864087936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573630.981482:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573630.981484:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573630.981486:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864087936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573631.428912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573631.492818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573632.452845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573632.516860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573633.476954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573633.540889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573634.500835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573634.564841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573635.524996:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573635.588874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573636.100868:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573636.100878:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573636.100882:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573636.100894:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5386 00000100:00080000:3.0:1788573636.100905:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573636.100915:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573636.100956:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864088064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573636.100988:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573636.100992:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573636.100994:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573636.100999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5386 00000100:00080000:3.0:1788573636.101003:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573636.101008:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573636.101019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573636.101042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864088192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573636.101052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573636.101056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573636.101057:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864088064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573636.101059:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573636.101071:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5386 00000100:00080000:3.0:1788573636.101078:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573636.101084:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573636.101093:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573636.101107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864088320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573636.101118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573636.101122:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573636.101125:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573636.101130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5386 00000100:00080000:3.0:1788573636.101133:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573636.101138:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788573636.101144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573636.101149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573636.101198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573636/real 1788573636] req@ffff92fc6733aa00 x1875449864088064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573636.101227:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864088064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573636.101245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573636.101246:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864088448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573636.101257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573636.101272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864088064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573636.101278:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573636.101284:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573636.101286:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573636.101311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573636.101334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573636.101365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573636/real 1788573636] req@ffff92fc67338e00 x1875449864088192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573636.101376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864088192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573636.101384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573636.101386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573636.101389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573636.101398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573636.101417:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573636.101425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573636/real 1788573636] req@ffff92fc6733a680 x1875449864088320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573636.101434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864088320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573636.101440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573636.101442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573636.101446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573636.101451:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088448:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788573636.101468:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5386 00000800:00000100:0.0:1788573636.101469:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573636.101480:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573636.101491:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573636/real 1788573636] req@ffff92fc67338000 x1875449864088448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573636.101503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864088448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573636.101510:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573636.101514:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573636.101517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088448:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788573636.101531:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573636.101542:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573636.101564:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864088576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573636.101622:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573636.101644:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573636.101653:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573636/real 1788573636] req@ffff92fc6733b480 x1875449864088576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573636.101661:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864088576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573636.101667:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573636.101669:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573636.101671:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573636.548907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573636.612865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573637.572925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573637.636857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573638.596944:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573638.660876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573639.620906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573639.684827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573640.644853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573640.708792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573641.220848:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.220858:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573641.220862:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.220872:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5391 00000100:00080000:0.0:1788573641.220881:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.220889:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573641.220935:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864088704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573641.220961:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.220964:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573641.220966:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.220971:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5391 00000100:00080000:0.0:1788573641.220975:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.220978:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573641.220987:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573641.220993:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864088704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573641.221000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864088832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573641.221009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.221012:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573641.221013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.221016:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5391 00000100:00080000:0.0:1788573641.221019:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.221021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573641.221025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573641.221039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864088960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573641.221054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573641.221062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.221066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573641.221069:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.221075:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5391 00000100:00080000:0.0:1788573641.221079:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573641.221082:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573641.221087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573641.221094:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff92fc67049880 x1875449864088704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573641.221099:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864089088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573641.221119:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573641.221121:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573641.221123:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573641.221126:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5391 00000100:00080000:0.0:1788573641.221129:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573641.221131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864088704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573641.221133:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573641.221138:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573641.221149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864089216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573641.221153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573641.221161:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573641.221166:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864088704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573641.221194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573641.221220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573641.221233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff92fc6704a300 x1875449864088832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573641.221242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864088832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573641.221249:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573641.221251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573641.221254:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573641.221259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573641.221288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573641.221300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff92fc6704aa00 x1875449864088960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573641.221309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864088960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573641.221315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573641.221317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573641.221320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864088960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573641.221324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573641.221338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573641.221344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff92fc6704a680 x1875449864089088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573641.221352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864089088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573641.221357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573641.221360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573641.221362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573641.221365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573641.221375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573641.221381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573641/real 1788573641] req@ffff92fc6704ad80 x1875449864089216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573641.221388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864089216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573641.221393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573641.221395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573641.221397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573641.668866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573641.732844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573642.692865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573642.756785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573643.716865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573643.780796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573644.740850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573644.804787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573645.764862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573645.828787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573646.340911:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573646.340920:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573646.340928:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573646.340939:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5396 00000100:00080000:3.0:1788573646.340958:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573646.340967:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573646.341006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864089344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573646.341031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573646.341034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573646.341036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573646.341039:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5396 00000100:00080000:3.0:1788573646.341042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573646.341045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573646.341053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573646.341063:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864089472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573646.341071:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573646.341074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573646.341075:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573646.341078:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5396 00000100:00080000:3.0:1788573646.341080:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573646.341084:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864089344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573646.341086:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573646.341093:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573646.341102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864089600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573646.341115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573646.341117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573646.341119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573646.341121:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5396 00000100:00080000:3.0:1788573646.341141:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573646.341143:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573646.341147:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573646.341152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573646.341158:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864089728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573646.341166:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573646.341169:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573646.341171:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573646.341173:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5396 00000100:00080000:3.0:1788573646.341176:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573646.341178:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573646.341181:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573646.341189:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864089856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573646.341214:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff92fc6733bb80 x1875449864089344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573646.341239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864089344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573646.341252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573646.341257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573646.341262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864089344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573646.341271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573646.341298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573646.341313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff92fc67338700 x1875449864089472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573646.341327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864089472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573646.341345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573646.341353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573646.341357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573646.341381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573646.341402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573646.341411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff92fc6733a300 x1875449864089600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573646.341423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864089600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573646.341431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573646.341435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573646.341438:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573646.341445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573646.341467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573646.341478:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff92fc67339500 x1875449864089728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573646.341487:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864089728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573646.341492:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573646.341494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573646.341496:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573646.341500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573646.341513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573646.341519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573646/real 1788573646] req@ffff92fc6733b800 x1875449864089856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573646.341526:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864089856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573646.341532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573646.341533:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573646.341535:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864089856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573646.788851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573646.852852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573647.812916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573647.876852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573648.836865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573648.900834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573649.860889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573649.924861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573650.884945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573650.948870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573651.460822:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573651.460831:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573651.460834:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573651.460843:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5402 00000100:00080000:1.0:1788573651.460852:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573651.460858:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573651.460890:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864089984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573651.460914:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573651.460917:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573651.460919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573651.460921:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5402 00000100:00080000:1.0:1788573651.460924:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573651.460927:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573651.460936:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573651.460950:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864090112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573651.460967:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864089984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573651.460974:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573651.460979:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573651.460981:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573651.460985:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5402 00000100:00080000:1.0:1788573651.460988:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573651.460991:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573651.460995:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573651.461007:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864090240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573651.461015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573651.461018:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573651.461019:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573651.461021:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5402 00000100:00080000:1.0:1788573651.461024:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573651.461026:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573651.461030:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573651.461038:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864090368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573651.461042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573651.461050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573651.461053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573651.461055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573651.461058:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5402 00000100:00080000:1.0:1788573651.461063:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573651.461066:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573651.461068:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573651.461073:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864090496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573651.461096:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff92fc67289c00 x1875449864089984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573651.461111:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864089984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573651.461147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573651.461150:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573651.461153:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864089984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573651.461174:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573651.461193:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573651.461203:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff92fc67288380 x1875449864090112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573651.461211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864090112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573651.461217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573651.461219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573651.461221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573651.461225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573651.461236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573651.461242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff92fc6728b100 x1875449864090240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573651.461249:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864090240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573651.461255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573651.461256:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573651.461258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573651.461262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573651.461274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573651.461279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff92fc67288a80 x1875449864090368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573651.461287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864090368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573651.461293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573651.461294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573651.461296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573651.461299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573651.461309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573651.461315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573651/real 1788573651] req@ffff92fc67288700 x1875449864090496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573651.461322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864090496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573651.461327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573651.461329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573651.461331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573651.908859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573651.972824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573652.932838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573652.996858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573653.956983:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573654.020873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573654.980924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573655.044870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573656.004975:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573656.068830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573656.580903:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.580914:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573656.580919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.580930:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5407 00000100:00080000:3.0:1788573656.580942:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.580949:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573656.581018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864090624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573656.581044:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.581047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573656.581049:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.581052:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5407 00000100:00080000:3.0:1788573656.581055:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.581057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573656.581067:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573656.581087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864090752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573656.581095:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.581098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573656.581099:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.581102:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5407 00000100:00080000:3.0:1788573656.581104:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.581107:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573656.581110:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573656.581119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864090880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573656.581124:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864090624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573656.581131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.581135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573656.581137:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.581141:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5407 00000100:00080000:3.0:1788573656.581143:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.581145:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573656.581149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573656.581162:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864091008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573656.581168:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573656.581171:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573656.581172:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573656.581178:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5407 00000100:00080000:3.0:1788573656.581189:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573656.581194:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788573656.581203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573656.581204:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573656.581218:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864091136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573656.581254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff92fc67338a80 x1875449864090624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573656.581276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864090624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573656.581286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573656.581289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573656.581292:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864090624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573656.581298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573656.581314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573656.581323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff92fc67338380 x1875449864090752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573656.581331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864090752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573656.581339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573656.581342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573656.581346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573656.581373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573656.581395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573656.581405:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff92fc6733b100 x1875449864090880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573656.581413:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864090880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573656.581423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573656.581425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573656.581428:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864090880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573656.581432:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573656.581444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573656.581450:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff92fc6733ad80 x1875449864091008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573656.581458:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864091008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573656.581463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573656.581464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573656.581466:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573656.581470:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573656.581481:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573656.581487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff92fc67339f80 x1875449864091136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573656.581495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864091136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573656.581499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573656.581501:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573656.581504:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573657.028925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573657.092858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573658.052988:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573658.116849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573659.076862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573659.140833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573660.100916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573660.164809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573661.124862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573661.188833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573661.700805:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.700815:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573661.700820:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.700831:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5412 00000100:00080000:3.0:1788573661.700842:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.700849:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573661.700894:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864091264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.700924:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.700929:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573661.700931:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.700936:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5412 00000100:00080000:3.0:1788573661.700940:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.700945:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573661.700955:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573661.700966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864091392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.700973:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.700976:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573661.700978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.700980:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5412 00000100:00080000:3.0:1788573661.700983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.700987:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573661.700990:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864091264:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573661.700991:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573661.701000:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864091520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.701007:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.701009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573661.701011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.701014:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5412 00000100:00080000:3.0:1788573661.701016:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.701018:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573661.701022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573661.701030:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864091648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573661.701037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573661.701039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573661.701040:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573661.701042:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5412 00000100:00080000:3.0:1788573661.701046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573661.701048:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573661.701051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573661.701058:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864091776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573661.701108:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573661.701151:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573661.701178:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573661.701205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573662.148828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573662.212794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788573663.172953:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573663.236858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573664.196961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573664.260801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788573664.773191:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788573664.773240:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573664.773258:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573664.773299:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573664.773309:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573664.773318:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573664.773327:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788573664.773346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff92fc67339180 x1875449864091520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.773374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864091520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.773392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.773398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573664.773402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573664.773410:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff92fc67339f80 x1875449864091648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.773423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864091648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.773430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.773433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573664.773436:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573664.773443:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff92fc6733ad80 x1875449864091776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.773454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864091776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.773461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.773464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573664.773467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091776:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573664.773473:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff92fc67339c00 x1875449864091264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.773483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864091264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.773491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.773494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573664.773497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864091264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573664.773504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573664] req@ffff92fc67339880 x1875449864091392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573664.773515:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864091392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573664.773523:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573664.773526:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573664.773529:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864091392:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788573665.221056:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573665.284879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573666.244938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573666.308890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573666.820992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.821000:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573666.821004:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.821013:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5417 00000100:00080000:2.0:1788573666.821023:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.821029:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573666.821060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864091904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573666.821082:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.821086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573666.821087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.821090:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5417 00000100:00080000:2.0:1788573666.821092:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.821094:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573666.821104:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573666.821118:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864092032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573666.821130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.821140:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573666.821142:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.821148:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5417 00000100:00080000:2.0:1788573666.821156:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573666.821158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864091904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573666.821163:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573666.821177:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573666.821195:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864092160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573666.821207:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.821211:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573666.821213:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.821218:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5417 00000100:00080000:2.0:1788573666.821225:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.821229:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573666.821237:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573666.821253:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864092288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573666.821263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573666.821277:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.821281:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573666.821284:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.821290:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5417 00000100:00080000:2.0:1788573666.821295:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.821301:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573666.821306:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573666.821319:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864092416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573666.821352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff92fc4617f480 x1875449864091904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.821383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864091904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.821398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.821403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573666.821409:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864091904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573666.821420:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573666.821452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573666.821468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff92fc4617ed80 x1875449864092032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.821483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864092032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.821494:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.821498:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573666.821501:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573666.821543:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573666.821577:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573666.821593:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff92fc4617f100 x1875449864092160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.821608:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864092160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.821618:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.821622:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573666.821626:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573666.821634:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573666.821658:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573666.821671:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff92fc4617f800 x1875449864092288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.821685:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864092288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.821693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.821696:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573666.821700:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573666.821707:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573666.821798:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573666.821817:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff92fc4617c700 x1875449864092416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.821830:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864092416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.821838:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.821842:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573666.821845:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573667.268864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573667.332855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573668.292878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573668.356830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573669.316818:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573669.380838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573670.340908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573670.404830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573671.364833:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573671.428835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573671.940907:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.940918:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573671.940923:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.940939:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5422 00000100:00080000:0.0:1788573671.940955:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.940964:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573671.941022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864092544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573671.941056:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.941059:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573671.941061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.941064:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5422 00000100:00080000:0.0:1788573671.941067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.941070:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573671.941081:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573671.941093:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864092672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573671.941101:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.941104:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573671.941106:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.941109:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5422 00000100:00080000:0.0:1788573671.941112:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.941114:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573671.941119:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573671.941129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864092800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573671.941152:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864092544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573671.941158:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.941162:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573671.941163:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.941167:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5422 00000100:00080000:0.0:1788573671.941170:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.941173:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573671.941177:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573671.941190:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864092928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573671.941200:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.941205:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573671.941207:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.941212:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5422 00000100:00080000:0.0:1788573671.941217:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.941222:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573671.941228:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573671.941242:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864093056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573671.941263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573671.941344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff92fc6704b100 x1875449864092544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573671.941371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864092544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573671.941385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573671.941390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573671.941395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864092544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573671.941404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573671.941434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573671.941451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff92fc67049500 x1875449864092672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573671.941466:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864092672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573671.941478:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573671.941482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573671.941486:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573671.941530:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573671.941560:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573671.941577:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff92fc67049180 x1875449864092800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573671.941595:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864092800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573671.941623:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573671.941629:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573671.941635:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573671.941660:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573671.941689:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573671.941703:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff92fc67049c00 x1875449864092928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573671.941725:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864092928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573671.941782:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573671.941788:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573671.941792:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864092928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573671.941799:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573671.941825:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573671.941839:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff92fc67048e00 x1875449864093056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573671.941852:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864093056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573671.941862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573671.941865:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573671.941869:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573672.388982:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573672.452887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573673.412835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573673.476798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573674.436878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573674.500841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573675.460866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573675.524793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573676.484977:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573676.548810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573677.060888:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573677.060903:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573677.060909:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573677.060923:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5427 00000100:00080000:0.0:1788573677.060937:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573677.060946:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573677.060999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864093184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573677.061043:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573677.061047:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573677.061049:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573677.061054:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5427 00000100:00080000:0.0:1788573677.061058:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573677.061062:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573677.061076:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573677.061094:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864093312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573677.061107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573677.061112:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573677.061112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864093184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573677.061115:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573677.061121:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5427 00000100:00080000:0.0:1788573677.061127:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573677.061134:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573677.061141:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573677.061156:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864093440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573677.061167:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573677.061171:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573677.061174:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573677.061178:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5427 00000100:00080000:0.0:1788573677.061197:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573677.061202:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573677.061209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788573677.061210:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573677.061228:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864093568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573677.061240:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573677.061244:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573677.061246:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573677.061251:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5427 00000100:00080000:0.0:1788573677.061257:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573677.061262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573677/real 1788573677] req@ffff92fc6704bb80 x1875449864093184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573677.061265:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573677.061275:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573677.061295:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864093696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573677.061298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864093184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573677.061327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573677.061332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573677.061337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864093184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573677.061361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573677.061380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573677.061390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573677/real 1788573677] req@ffff92fc67048e00 x1875449864093312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573677.061401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864093312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573677.061416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573677.061419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573677.061421:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573677.061425:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573677.061438:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573677.061445:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573677/real 1788573677] req@ffff92fc67049c00 x1875449864093440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573677.061454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864093440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573677.061464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573677.061466:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573677.061469:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573677.061488:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573677.061503:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573677.061511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573677/real 1788573677] req@ffff92fc67049180 x1875449864093568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573677.061524:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864093568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573677.061530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573677.061532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573677.061534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573677.061540:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573677.061559:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573677.061571:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573677/real 1788573677] req@ffff92fc67049500 x1875449864093696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573677.061584:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864093696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573677.061593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573677.061596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573677.061599:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573677.508864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573677.572812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573678.532940:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573678.596879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573679.556896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573679.620798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573680.580893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573680.644837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573681.604857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573681.668784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573682.180844:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573682.180852:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573682.180856:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573682.180865:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5432 00000100:00080000:3.0:1788573682.180874:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573682.180880:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573682.180911:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864093824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573682.180937:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573682.180941:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573682.180943:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573682.180948:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5432 00000100:00080000:3.0:1788573682.180951:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573682.180955:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573682.180966:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573682.180978:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864093952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573682.180984:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864093824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573682.180989:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573682.180997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573682.180999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573682.181002:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5432 00000100:00080000:3.0:1788573682.181005:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573682.181009:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573682.181013:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573682.181023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864094080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573682.181031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573682.181034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573682.181036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573682.181039:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5432 00000100:00080000:3.0:1788573682.181042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573682.181045:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573682.181048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573682.181049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573682.181059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864094208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573682.181068:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573682.181071:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573682.181072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573682.181076:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5432 00000100:00080000:3.0:1788573682.181079:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573682.181082:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573682.181086:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573682.181088:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff92fc6733b100 x1875449864093824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573682.181094:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864094336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573682.181106:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864093824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.181116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.181120:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573682.181124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864093824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573682.181146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573682.181166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573682.181176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff92fc67338380 x1875449864093952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.181184:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864093952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.181190:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.181192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573682.181195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864093952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573682.181198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573682.181210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573682.181216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff92fc67338a80 x1875449864094080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.181223:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864094080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.181228:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.181230:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573682.181232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573682.181235:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573682.181246:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573682.181251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff92fc6733b800 x1875449864094208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.181258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864094208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.181263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.181265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573682.181266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573682.181269:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573682.181279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573682.181284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573682/real 1788573682] req@ffff92fc67339500 x1875449864094336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573682.181296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864094336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573682.181301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573682.181303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573682.181305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573682.628879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573682.692831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573683.652923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573683.716897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573684.676928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573684.740818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573685.700906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573685.764824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573686.724878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573686.788837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573687.300826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573687.300837:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573687.300841:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573687.300853:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5437 00000100:00080000:3.0:1788573687.300864:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573687.300872:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573687.300906:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864094464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573687.300933:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573687.300938:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573687.300940:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573687.300943:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5437 00000100:00080000:3.0:1788573687.300946:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573687.300949:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573687.300957:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573687.300967:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864094592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573687.300977:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864094464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573687.300978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573687.300985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573687.300988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573687.300991:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5437 00000100:00080000:3.0:1788573687.300993:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573687.300995:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573687.300999:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573687.301008:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864094720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573687.301014:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573687.301016:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573687.301018:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573687.301020:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5437 00000100:00080000:3.0:1788573687.301024:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573687.301026:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573687.301030:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573687.301036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864094848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573687.301043:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573687.301045:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573687.301046:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573687.301048:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5437 00000100:00080000:3.0:1788573687.301050:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573687.301054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573687.301056:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573687.301061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573687.301067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864094976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573687.301105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff92fc6733a300 x1875449864094464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.301121:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864094464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.301129:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.301132:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573687.301135:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864094464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573687.301155:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573687.301173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573687.301184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff92fc67338700 x1875449864094592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.301204:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864094592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.301227:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.301232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573687.301236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573687.301242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573687.301263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573687.301275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff92fc6733bb80 x1875449864094720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.301288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864094720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.301297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.301300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573687.301304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573687.301311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573687.301326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573687.301334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff92fc6733b480 x1875449864094848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.301341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864094848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.301352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.301354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573687.301356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573687.301360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573687.301370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573687.301375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573687/real 1788573687] req@ffff92fc67338000 x1875449864094976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573687.301382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864094976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573687.301387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573687.301389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573687.301391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864094976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573687.748848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573687.812846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573688.772839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573688.836840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573689.796856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573689.860855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573690.820841:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573690.884812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573691.844894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573691.908913:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573692.420786:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573692.420796:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573692.420801:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573692.420811:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5443 00000100:00080000:3.0:1788573692.420822:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573692.420831:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573692.420878:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864095104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573692.420903:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573692.420907:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573692.420908:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573692.420911:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5443 00000100:00080000:3.0:1788573692.420914:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573692.420917:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573692.420928:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573692.420938:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864095232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573692.420947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573692.420949:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573692.420951:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573692.420955:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5443 00000100:00080000:3.0:1788573692.420959:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573692.420963:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573692.420964:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864095104:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573692.420967:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573692.420979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864095360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573692.420985:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573692.420988:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573692.420989:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573692.420992:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5443 00000100:00080000:3.0:1788573692.420994:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573692.420996:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573692.421001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573692.421010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864095488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573692.421022:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573692.421024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573692.421026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573692.421031:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5443 00000100:00080000:3.0:1788573692.421033:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573692.421035:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573692.421038:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573692.421046:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864095616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573692.421058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573692.421135:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff92fc6733a680 x1875449864095104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.421157:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864095104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.421166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.421170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573692.421173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864095104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573692.421201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573692.421221:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573692.421231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff92fc67338e00 x1875449864095232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.421239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864095232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.421246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.421248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573692.421251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573692.421254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573692.421266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573692.421273:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff92fc6733aa00 x1875449864095360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.421280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864095360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.421286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.421288:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573692.421289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573692.421293:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573692.421305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573692.421311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff92fc67338000 x1875449864095488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.421319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864095488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.421324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.421326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573692.421329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573692.421332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573692.421342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573692.421348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573692/real 1788573692] req@ffff92fc6733b480 x1875449864095616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573692.421355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864095616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573692.421359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573692.421362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573692.421364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573692.868989:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573692.932859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573693.892867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573693.956846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573694.916951:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573694.980827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573695.940863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573696.004864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573696.964936:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573697.028821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573697.540842:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573697.540854:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573697.540857:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573697.540866:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5448 00000100:00080000:3.0:1788573697.540875:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573697.540880:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573697.540912:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864095744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573697.540932:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573697.540935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573697.540937:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573697.540940:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5448 00000100:00080000:3.0:1788573697.540942:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573697.540944:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573697.540953:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573697.540974:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864095872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573697.540982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573697.540985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573697.540986:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573697.540989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5448 00000100:00080000:3.0:1788573697.540991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573697.540994:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573697.540997:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573697.541005:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864095744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573697.541006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864096000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573697.541020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573697.541023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573697.541025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573697.541029:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5448 00000100:00080000:3.0:1788573697.541033:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573697.541036:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573697.541042:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573697.541055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864096128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573697.541066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573697.541070:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573697.541073:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573697.541080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5448 00000800:00000100:1.0:1788573697.541082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573697.541085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573697.541091:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573697.541097:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573697.541110:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864096256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573697.541133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff92fc6733bb80 x1875449864095744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573697.541154:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864095744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573697.541164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573697.541169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573697.541173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864095744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573697.541181:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573697.541205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573697.541219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff92fc67338700 x1875449864095872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573697.541232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864095872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573697.541241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573697.541244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573697.541248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864095872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573697.541272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573697.541294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573697.541307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff92fc6733a300 x1875449864096000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573697.541320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864096000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573697.541335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573697.541338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573697.541342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573697.541348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573697.541369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573697.541381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff92fc67339500 x1875449864096128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573697.541394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864096128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573697.541404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573697.541407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573697.541411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573697.541418:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573697.541438:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573697.541451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573697/real 1788573697] req@ffff92fc6733b800 x1875449864096256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573697.541464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864096256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573697.541472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573697.541476:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573697.541479:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573697.988916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573698.052822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573699.012880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573699.076796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573700.036936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573700.100866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573701.060979:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573701.124807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573702.084914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573702.148865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573702.660906:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573702.660917:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573702.660922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573702.660932:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5453 00000100:00080000:3.0:1788573702.660945:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573702.660953:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573702.660999:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864096384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573702.661030:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573702.661034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573702.661035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573702.661038:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5453 00000100:00080000:3.0:1788573702.661041:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573702.661043:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573702.661053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573702.661065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864096512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573702.661079:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573702.661082:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573702.661083:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573702.661088:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5453 00000100:00080000:3.0:1788573702.661091:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573702.661092:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864096384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573702.661096:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573702.661103:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573702.661110:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864096640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573702.661117:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573702.661120:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573702.661121:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573702.661124:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5453 00000100:00080000:3.0:1788573702.661126:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573702.661128:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573702.661131:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573702.661139:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864096768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573702.661144:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573702.661146:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573702.661147:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573702.661149:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5453 00000100:00080000:3.0:1788573702.661154:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573702.661156:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573702.661159:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573702.661165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573702.661169:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864096896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573702.661216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff92fc67338a80 x1875449864096384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573702.661234:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864096384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573702.661243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573702.661247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573702.661250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864096384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573702.661273:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573702.661289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573702.661298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff92fc67338380 x1875449864096512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573702.661307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864096512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573702.661313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573702.661316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573702.661320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573702.661326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573702.661345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573702.661357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff92fc6733b100 x1875449864096640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573702.661371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864096640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573702.661377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573702.661379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573702.661381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573702.661386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573702.661403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573702.661413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff92fc67339880 x1875449864096768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573702.661424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864096768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573702.661433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573702.661437:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573702.661441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573702.661448:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573702.661467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573702.661475:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff92fc67339c00 x1875449864096896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573702.661488:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864096896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573702.661494:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573702.661496:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573702.661498:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864096896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573703.108922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573703.172852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573704.132919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573704.196839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573705.156910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573705.220873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573706.180891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573706.244823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573707.204922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573707.268875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573707.780824:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573707.780835:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573707.780840:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573707.780858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5458 00000100:00080000:0.0:1788573707.780871:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573707.780879:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573707.780931:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864097024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573707.780965:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573707.780969:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573707.780971:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573707.780974:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5458 00000100:00080000:0.0:1788573707.780978:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573707.780983:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573707.780994:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573707.781002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864097024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573707.781010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864097152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573707.781020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573707.781023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573707.781025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573707.781030:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5458 00000100:00080000:0.0:1788573707.781035:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573707.781040:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573707.781045:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573707.781059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864097280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573707.781066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573707.781069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573707.781071:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573707.781073:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5458 00000100:00080000:0.0:1788573707.781077:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573707.781079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788573707.781083:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573707.781088:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573707.781099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864097408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573707.781106:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573707.781109:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573707.781110:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573707.781112:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5458 00000100:00080000:0.0:1788573707.781116:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573707.781119:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573707.781124:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573707.781130:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff92fc6704b100 x1875449864097024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573707.781134:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864097536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573707.781220:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864097024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573707.781230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573707.781234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573707.781237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864097024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573707.781261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573707.781288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573707.781299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff92fc6704ad80 x1875449864097152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573707.781307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864097152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573707.781314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573707.781316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573707.781320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573707.781324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788573707.781336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573707.781466:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff92fc6704a680 x1875449864097280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573707.781476:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864097280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573707.781483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573707.781486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573707.781489:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573707.781524:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573707.781555:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573707.781582:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff92fc6704aa00 x1875449864097408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573707.781597:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864097408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573707.781603:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573707.781605:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573707.781607:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788573707.781612:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788573707.781635:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573707.781645:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff92fc6704a300 x1875449864097536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573707.781652:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864097536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573707.781658:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573707.781660:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573707.781662:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573708.228862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573708.292885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573709.252913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573709.316793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573710.276833:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573710.340857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573711.300891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573711.364834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573712.324852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573712.388822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573712.900829:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.900838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573712.900842:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.900851:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5463 00000100:00080000:3.0:1788573712.900861:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.900867:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573712.900900:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864097664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.900926:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.900929:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573712.900931:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.900933:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5463 00000100:00080000:3.0:1788573712.900936:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.900938:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573712.900946:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573712.900955:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864097792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.900963:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.900968:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573712.900969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864097664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573712.900970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.900977:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5463 00000100:00080000:3.0:1788573712.900980:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.900982:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573712.900985:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573712.900994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864097920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.901002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.901004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573712.901006:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.901008:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5463 00000100:00080000:3.0:1788573712.901010:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.901012:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573712.901015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573712.901023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864098048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.901028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.901030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573712.901031:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.901034:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5463 00000100:00080000:3.0:1788573712.901037:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.901039:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573712.901042:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573712.901047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573712.901052:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864098176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573712.901087:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff92fc6733ad80 x1875449864097664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573712.901103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864097664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573712.901112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573712.901115:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573712.901117:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864097664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573712.901137:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573712.901152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573712.901161:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff92fc67339f80 x1875449864097792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573712.901170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864097792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573712.901176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573712.901179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573712.901181:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573712.901184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573712.901195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573712.901201:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff92fc67339180 x1875449864097920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573712.901209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864097920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573712.901216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573712.901219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573712.901222:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864097920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573712.901228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573712.901248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573712.901260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff92fc67339c00 x1875449864098048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573712.901273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864098048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573712.901285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573712.901287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573712.901290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573712.901294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573712.901309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573712.901317:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff92fc67339880 x1875449864098176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573712.901324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864098176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573712.901329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573712.901331:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573712.901333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573713.348901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573713.412839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573714.372846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573714.436782:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573715.396909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573715.460817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573716.420851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573716.484794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573717.444877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573717.508833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573718.020875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573718.020886:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573718.020890:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573718.020901:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5468 00000100:00080000:2.0:1788573718.020912:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573718.020919:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573718.020955:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864098304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573718.020982:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573718.020986:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573718.020988:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573718.020992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5468 00000100:00080000:2.0:1788573718.020996:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573718.021000:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573718.021009:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573718.021022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864098432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573718.021033:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573718.021037:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864098304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573718.021038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573718.021051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573718.021056:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5468 00000100:00080000:2.0:1788573718.021060:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573718.021064:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573718.021070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573718.021082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864098560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573718.021091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573718.021094:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573718.021097:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573718.021100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5468 00000100:00080000:2.0:1788573718.021104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573718.021108:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573718.021115:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788573718.021117:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573718.021131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864098688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573718.021147:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573718.021150:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573718.021152:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573718.021157:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5468 00000100:00080000:2.0:1788573718.021161:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573718.021165:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573718/real 1788573718] req@ffff92fc4617ca80 x1875449864098304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573718.021168:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573718.021177:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573718.021193:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864098816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573718.021199:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864098304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573718.021222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573718.021227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573718.021233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864098304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573718.021261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573718.021289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573718.021303:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573718/real 1788573718] req@ffff92fc4617dc00 x1875449864098432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573718.021317:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864098432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573718.021326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573718.021330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573718.021333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573718.021339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573718.021363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573718.021376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573718/real 1788573718] req@ffff92fc4617e680 x1875449864098560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573718.021388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864098560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573718.021397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573718.021401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573718.021404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573718.021411:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573718.021435:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573718.021447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573718/real 1788573718] req@ffff92fc4617d180 x1875449864098688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573718.021475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864098688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573718.021485:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573718.021489:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573718.021493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573718.021510:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573718.021531:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573718.021543:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573718/real 1788573718] req@ffff92fc4617c380 x1875449864098816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573718.021555:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864098816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573718.021564:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573718.021567:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573718.021580:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864098816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573718.468860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573718.532787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573719.492887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573719.556829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573720.516868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573720.580781:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573721.540844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573721.604840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573722.564887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573722.628791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573723.140850:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573723.140858:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573723.140861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573723.140870:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5473 00000100:00080000:3.0:1788573723.140879:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573723.140885:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573723.140919:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864098944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573723.140941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573723.140945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573723.140946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573723.140948:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5473 00000100:00080000:3.0:1788573723.140951:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573723.140953:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573723.140960:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573723.140969:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864099072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573723.140977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573723.140981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573723.140983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573723.140986:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5473 00000100:00080000:3.0:1788573723.140989:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573723.140992:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864098944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573723.140994:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573723.141007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573723.141016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864099200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573723.141023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573723.141025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573723.141026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573723.141029:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5473 00000100:00080000:3.0:1788573723.141030:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573723.141032:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573723.141035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573723.141043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864099328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573723.141049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573723.141051:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573723.141052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573723.141054:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5473 00000100:00080000:3.0:1788573723.141056:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573723.141059:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573723.141063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573723.141069:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864099456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573723.141096:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573723.141140:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573723/real 1788573723] req@ffff92fc6733b100 x1875449864098944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573723.141160:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864098944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573723.141171:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573723.141176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573723.141180:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864098944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573723.141188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573723.141222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573723.141236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573723/real 1788573723] req@ffff92fc67338380 x1875449864099072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573723.141249:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864099072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573723.141259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573723.141262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573723.141266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573723.141312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573723.141334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573723.141346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573723/real 1788573723] req@ffff92fc67338a80 x1875449864099200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573723.141417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864099200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573723.141426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573723.141430:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573723.141433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573723.141439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573723.141470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573723.141482:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573723/real 1788573723] req@ffff92fc6733b800 x1875449864099328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573723.141495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864099328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573723.141503:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573723.141506:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573723.141510:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573723.141516:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573723.141533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573723.141544:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573723/real 1788573723] req@ffff92fc67339500 x1875449864099456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573723.141556:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864099456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573723.141570:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573723.141573:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573723.141576:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573723.588944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573723.652851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573724.612899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573724.676866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573725.636901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573725.700860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573726.660927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573726.724881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573727.684859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573727.748841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573728.260823:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.260831:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573728.260834:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.260843:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5478 00000100:00080000:3.0:1788573728.260852:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.260858:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573728.260891:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864099584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573728.260916:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.260920:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573728.260922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.260926:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5478 00000100:00080000:3.0:1788573728.260930:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.260934:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573728.260944:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573728.260962:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864099712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573728.260976:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864099584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573728.260976:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.260985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573728.260988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.260992:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5478 00000100:00080000:3.0:1788573728.260996:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.260999:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573728.261004:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573728.261015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864099840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573728.261023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.261025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573728.261026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.261030:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5478 00000100:00080000:3.0:1788573728.261033:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.261035:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573728.261039:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573728.261047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864099968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573728.261053:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573728.261056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573728.261057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573728.261060:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5478 00000100:00080000:3.0:1788573728.261063:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573728.261066:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573728.261069:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573728.261076:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864100096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573728.261084:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573728.261129:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573728.261158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573728.261188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573728.708880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573728.772841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573729.732854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573729.796806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573730.756919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573730.820873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573731.332945:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573731.332981:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573731.332994:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573731.333031:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573731.333045:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573731.333054:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573731.333062:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573731.333071:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff92fc67338700 x1875449864099712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.333096:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864099712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.333109:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.333114:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573731.333118:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573731.333127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff92fc6733bb80 x1875449864099840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.333141:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864099840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.333162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.333166:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573731.333170:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573731.333177:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff92fc6733b480 x1875449864099968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.333226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864099968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.333236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.333241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573731.333244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864099968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573731.333251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff92fc6733a300 x1875449864099584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.333277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864099584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.333286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.333290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573731.333293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864099584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573731.333323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573728/real 1788573731] req@ffff92fc67338000 x1875449864100096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573731.333337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864100096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573731.333346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573731.333348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573731.333352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573731.780868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573731.844821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573732.804838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573732.868826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573733.380866:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573733.380874:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573733.380877:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573733.380886:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5483 00000100:00080000:0.0:1788573733.380894:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573733.380900:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573733.380933:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864100224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573733.380971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573733.380976:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573733.380978:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573733.380983:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5483 00000100:00080000:0.0:1788573733.380987:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573733.380991:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573733.381000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573733.381015:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864100352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573733.381027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864100224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573733.381029:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573733.381038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573733.381040:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573733.381045:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5483 00000100:00080000:0.0:1788573733.381050:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573733.381055:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573733.381059:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573733.381071:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864100480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573733.381078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573733.381081:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573733.381083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573733.381088:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5483 00000100:00080000:0.0:1788573733.381093:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573733.381097:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573733.381103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573733.381106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573733.381133:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864100608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573733.381145:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573733.381149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573733.381152:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573733.381173:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5483 00000100:00080000:0.0:1788573733.381178:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573733.381182:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573733.381187:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573733.381190:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff92fc67049880 x1875449864100224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573733.381202:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864100736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573733.381230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864100224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.381245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.381250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573733.381256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864100224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573733.381284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573733.381310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573733.381323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff92fc67048000 x1875449864100352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.381336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864100352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.381352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.381356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573733.381359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573733.381366:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573733.381387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573733.381398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff92fc67049f80 x1875449864100480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.381410:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864100480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.381418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.381422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573733.381425:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573733.381431:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573733.381451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573733.381463:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff92fc67048700 x1875449864100608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.381475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864100608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.381483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.381487:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573733.381491:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573733.381498:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573733.381517:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573733.381527:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573733/real 1788573733] req@ffff92fc67048380 x1875449864100736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573733.381538:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864100736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573733.381546:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573733.381549:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573733.381552:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573733.828916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573733.892856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573734.852853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573734.916833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573735.876912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573735.940791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573736.900889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573736.964867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573737.924921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573737.988797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573738.500878:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.500889:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573738.500894:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.500921:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5489 00000100:00080000:3.0:1788573738.500934:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.500942:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573738.500989:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864100864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.501039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.501043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573738.501045:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.501048:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5489 00000100:00080000:3.0:1788573738.501051:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.501096:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573738.501108:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573738.501119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864100992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.501147:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.501150:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573738.501151:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.501156:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5489 00000100:00080000:3.0:1788573738.501162:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573738.501164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864100864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573738.501168:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573738.501183:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573738.501195:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864101120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.501204:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.501207:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573738.501208:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.501211:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5489 00000100:00080000:3.0:1788573738.501214:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.501217:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573738.501221:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573738.501231:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864101248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573738.501238:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573738.501240:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573738.501242:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573738.501246:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5489 00000800:00000100:1.0:1788573738.501252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573738.501253:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573738.501263:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573738.501267:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573738.501279:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864101376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573738.501320:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff92fc6733aa00 x1875449864100864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.501336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864100864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.501345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.501348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573738.501351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864100864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573738.501373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573738.501389:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573738.501398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff92fc67338e00 x1875449864100992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.501406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864100992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.501412:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.501414:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573738.501417:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864100992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573738.501421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573738.501432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573738.501438:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff92fc6733a680 x1875449864101120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.501445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864101120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.501450:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.501452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573738.501454:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573738.501458:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573738.501470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573738.501476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff92fc67338000 x1875449864101248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.501485:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864101248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.501489:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.501491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573738.501493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573738.501497:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573738.501507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573738.501513:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573738/real 1788573738] req@ffff92fc6733a300 x1875449864101376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573738.501520:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864101376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573738.501524:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573738.501526:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573738.501528:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573738.948907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573739.012900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573739.973006:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573740.036844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573740.996850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573741.060823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573742.020883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573742.084820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573743.044836:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573743.108833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573743.620828:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.620841:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573743.620845:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.620854:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5494 00000100:00080000:3.0:1788573743.620870:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.620879:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573743.620915:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864101504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.620948:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.620964:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.620968:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.620973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5494 00000100:00080000:3.0:1788573743.620978:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.620984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573743.620995:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573743.621017:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864101632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573743.621018:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864101504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573743.621037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.621041:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573743.621044:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.621048:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5494 00000100:00080000:3.0:1788573743.621052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.621057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573743.621063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573743.621078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864101760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.621095:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.621099:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.621102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573743.621105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573743.621110:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5494 00000100:00100000:2.0:1788573743.621151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff92fc6733b480 x1875449864101504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573743.621175:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864101504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573743.621181:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.621192:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788573743.621196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573743.621199:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573743.621204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573743.621210:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864101504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573743.621219:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864101888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.621230:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.621233:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573743.621235:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573743.621238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101632:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573743.621242:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5494 00000800:00000100:2.0:1788573743.621267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573743.621281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff92fc6733bb80 x1875449864101632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573743.621296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864101632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573743.621306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573743.621312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573743.621316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573743.621322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573743.621345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573743.621360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff92fc67338700 x1875449864101760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573743.621361:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.621371:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573743.621385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864101760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573743.621396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573743.621400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573743.621404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573743.621414:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101888:192.168.202.132@tcp:8:kworker.0 00000008:00100000:3.0:1788573743.621421:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573743.621441:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864102016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573743.621449:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573743.621470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff92fc67339500 x1875449864101888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573743.621485:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864101888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573743.621512:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573743.621515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573743.621519:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864101888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573743.621536:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573743.621557:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573743.621569:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573743] req@ffff92fc6733b800 x1875449864102016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573743.621582:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864102016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573743.621591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573743.621594:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573743.621598:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573744.068905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573744.132814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573745.092866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573745.156833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573746.116933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573746.180826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573747.140841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573747.204835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573748.164959:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573748.228809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573748.740780:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.740788:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573748.740792:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.740800:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5499 00000100:00080000:3.0:1788573748.740810:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.740818:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573748.740879:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864102144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573748.740905:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.740908:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573748.740909:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.740912:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5499 00000100:00080000:3.0:1788573748.740916:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.740919:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573748.740929:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573748.740939:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864102144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573748.740956:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864102272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573748.740971:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.740975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573748.740977:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.740981:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5499 00000100:00080000:3.0:1788573748.740985:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.740989:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573748.740994:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573748.740998:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573748.741007:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864102400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573748.741015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.741018:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573748.741019:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573748.741022:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5499 00000100:00080000:3.0:1788573748.741025:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573748.741028:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573748.741031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573748.741039:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864102528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573748.741043:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff92fc67338a80 x1875449864102144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.741064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573748.741069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573748.741071:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573748.741074:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864102144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.741078:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5499 00000100:00080000:3.0:1788573748.741085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573748.741090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573748.741090:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788573748.741099:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:3.0:1788573748.741099:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573748.741106:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864102144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573748.741113:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864102656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573748.741129:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573748.741148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573748.741157:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff92fc67338380 x1875449864102272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573748.741165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864102272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573748.741172:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573748.741174:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573748.741177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573748.741180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573748.741191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573748.741197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff92fc6733b100 x1875449864102400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573748.741212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864102400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573748.741218:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573748.741220:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573748.741222:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573748.741225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573748.741237:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573748.741243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff92fc67339880 x1875449864102528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573748.741251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864102528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573748.741256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573748.741258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573748.741261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573748.741264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573748.741273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573748.741279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff92fc67339c00 x1875449864102656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573748.741286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864102656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573748.741291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573748.741293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573748.741295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573749.188834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573749.252829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573750.212875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573750.276863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573751.236856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573751.300848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573752.260914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573752.324801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573753.284878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573753.348885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573753.860833:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573753.860845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573753.860850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573753.860860:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5504 00000100:00080000:2.0:1788573753.860870:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573753.860876:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573753.860907:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864102784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573753.860934:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573753.860938:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573753.860939:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573753.860942:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5504 00000100:00080000:2.0:1788573753.860944:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573753.860946:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573753.860954:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573753.860964:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864102912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573753.860971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573753.860974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573753.860975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573753.860977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5504 00000100:00080000:2.0:1788573753.860979:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573753.860982:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573753.860987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864102784:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788573753.860987:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573753.860998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864103040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573753.861005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573753.861008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573753.861009:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573753.861012:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5504 00000100:00080000:2.0:1788573753.861015:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573753.861017:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573753.861021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573753.861029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864103168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573753.861035:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573753.861037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573753.861038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573753.861040:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5504 00000100:00080000:2.0:1788573753.861043:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573753.861045:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573753.861048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573753.861054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573753.861057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864103296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573753.861095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff92fc4617fb80 x1875449864102784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.861110:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864102784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.861118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.861121:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573753.861124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864102784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573753.861129:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573753.861143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573753.861151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff92fc4617c000 x1875449864102912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.861159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864102912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.861165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.861166:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573753.861168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864102912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573753.861188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573753.861198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573753.861205:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff92fc4617d880 x1875449864103040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.861217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864103040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.861222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.861223:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573753.861225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573753.861229:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573753.861239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573753.861245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff92fc4617e300 x1875449864103168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.861252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864103168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.861257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.861258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573753.861260:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573753.861264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573753.861273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573753.861278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff92fc4617ea00 x1875449864103296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.861285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864103296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573753.861294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573753.861296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573753.861299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573754.308953:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573754.372853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573755.332861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573755.396821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573756.356939:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573756.420902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573757.380912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573757.444888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573758.404976:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573758.468884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573758.980846:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.980854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573758.980858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.980867:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5509 00000100:00080000:2.0:1788573758.980876:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.980883:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573758.980915:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864103424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573758.980938:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.980944:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573758.980945:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.980948:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5509 00000100:00080000:2.0:1788573758.980950:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.980952:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573758.980960:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573758.980969:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864103552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573758.980978:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.980982:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788573758.980984:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864103424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573758.980984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.980992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5509 00000100:00080000:2.0:1788573758.980995:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.980997:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573758.981000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573758.981011:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864103680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573758.981017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.981020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573758.981021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.981024:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5509 00000100:00080000:2.0:1788573758.981027:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.981029:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573758.981033:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573758.981041:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864103808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573758.981047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573758.981054:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573758.981057:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573758.981058:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573758.981062:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5509 00000100:00080000:2.0:1788573758.981064:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573758.981066:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573758.981069:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573758.981077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864103936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573758.981089:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff92fc4617ce00 x1875449864103424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.981103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864103424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.981111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.981114:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573758.981117:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864103424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573758.981134:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573758.981148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573758.981157:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff92fc4617d500 x1875449864103552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.981164:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864103552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.981174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.981185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573758.981187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573758.981194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573758.981207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573758.981214:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff92fc4617ea00 x1875449864103680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.981221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864103680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.981226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.981228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573758.981230:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573758.981234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573758.981244:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573758.981250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff92fc4617e300 x1875449864103808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.981257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864103808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.981262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.981264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573758.981266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573758.981270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573758.981280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573758.981285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff92fc4617d880 x1875449864103936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573758.981292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864103936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573758.981297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573758.981299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573758.981300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864103936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573759.428873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573759.492776:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573760.452895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573760.516830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573761.476924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573761.540822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573762.500861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573762.564828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573763.524879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573763.588832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573764.100797:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573764.100808:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573764.100813:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573764.100823:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5514 00000100:00080000:2.0:1788573764.100834:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573764.100842:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573764.100886:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864104064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573764.100912:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573764.100916:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573764.100917:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573764.100920:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5514 00000100:00080000:2.0:1788573764.100923:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573764.100926:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573764.100935:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573764.100945:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864104192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573764.100953:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573764.100956:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573764.100957:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573764.100960:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5514 00000100:00080000:2.0:1788573764.100962:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573764.100964:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573764.100967:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573764.100971:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864104064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573764.100977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864104320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573764.100984:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573764.100987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573764.100988:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573764.100991:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5514 00000100:00080000:2.0:1788573764.100992:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573764.100994:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573764.100998:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573764.101006:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864104448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573764.101016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573764.101019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573764.101021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573764.101023:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5514 00000100:00080000:2.0:1788573764.101025:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573764.101027:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573764.101030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573764.101039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864104576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573764.101047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573764.101092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573764/real 1788573764] req@ffff92fc4617c000 x1875449864104064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573764.101108:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864104064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573764.101117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573764.101120:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573764.101123:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864104064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573764.101129:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573764.101144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573764.101153:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573764/real 1788573764] req@ffff92fc4617fb80 x1875449864104192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573764.101161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864104192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573764.101167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573764.101168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573764.101170:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573764.101192:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573764.101204:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573764.101210:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573764/real 1788573764] req@ffff92fc4617c380 x1875449864104320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573764.101218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864104320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573764.101223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573764.101225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573764.101226:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573764.101230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573764.101241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573764.101247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573764/real 1788573764] req@ffff92fc4617d180 x1875449864104448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573764.101260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864104448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573764.101265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573764.101266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573764.101268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573764.101271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573764.101282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573764.101287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573764/real 1788573764] req@ffff92fc4617e680 x1875449864104576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573764.101295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864104576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573764.101300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573764.101302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573764.101304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573764.548941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573764.612861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573765.572924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573765.636903:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573766.596930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573766.660878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573767.620857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573767.684843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573768.644890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573768.708809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573769.220888:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573769.220897:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573769.220901:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573769.220910:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5519 00000100:00080000:0.0:1788573769.220920:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573769.220926:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573769.220982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864104704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573769.221015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573769.221020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573769.221022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573769.221026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5519 00000100:00080000:0.0:1788573769.221047:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573769.221052:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573769.221062:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573769.221079:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864104832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573769.221092:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573769.221113:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573769.221115:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573769.221120:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5519 00000100:00080000:0.0:1788573769.221124:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573769.221127:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573769.221132:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573769.221159:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864104960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573769.221159:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864104704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573769.221188:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573769.221191:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573769.221192:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573769.221212:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5519 00000100:00080000:0.0:1788573769.221214:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573769.221217:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573769.221220:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573769.221234:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864105088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573769.221243:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573769.221247:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573769.221249:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573769.221254:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5519 00000100:00080000:0.0:1788573769.221258:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573769.221264:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573769.221270:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573769.221282:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864105216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573769.221304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573769.221370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff92fc67048a80 x1875449864104704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573769.221400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864104704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573769.221411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573769.221415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573769.221419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864104704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573769.221427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573769.221465:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573769.221479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff92fc67048380 x1875449864104832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573769.221489:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864104832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573769.221512:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573769.221515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573769.221517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573769.221543:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573769.221558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573769.221565:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff92fc67048700 x1875449864104960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573769.221573:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864104960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573769.221584:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573769.221587:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573769.221589:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864104960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573769.221593:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573769.221605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573769.221611:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff92fc67049f80 x1875449864105088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573769.221618:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864105088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573769.221624:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573769.221626:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573769.221630:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573769.221637:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573769.221657:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573769.221668:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573769/real 1788573769] req@ffff92fc67048000 x1875449864105216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573769.221681:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864105216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573769.221689:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573769.221692:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573769.221696:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573769.668865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573769.732843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573770.692870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573770.756794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573771.716873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573771.780805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573772.740877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573772.804820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573773.764876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573773.828838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573774.340843:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573774.340851:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573774.340855:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573774.340864:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5524 00000100:00080000:3.0:1788573774.340873:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573774.340880:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573774.340918:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864105344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573774.340948:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573774.340951:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573774.340953:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573774.340956:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5524 00000100:00080000:3.0:1788573774.340959:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573774.340962:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573774.340970:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573774.340982:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864105472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573774.340990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573774.340993:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573774.340994:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573774.340996:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5524 00000100:00080000:3.0:1788573774.340999:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573774.341004:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864105344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573774.341004:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573774.341010:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573774.341018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864105600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573774.341025:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573774.341028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573774.341029:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573774.341031:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5524 00000100:00080000:3.0:1788573774.341034:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573774.341037:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573774.341041:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573774.341049:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864105728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573774.341056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573774.341058:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573774.341060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573774.341061:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5524 00000100:00080000:3.0:1788573774.341064:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573774.341066:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573774.341069:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573774.341075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864105856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573774.341083:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573774.341132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff92fc67339180 x1875449864105344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573774.341154:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864105344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573774.341166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573774.341171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573774.341176:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864105344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573774.341184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573774.341210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573774.341225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff92fc67339f80 x1875449864105472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573774.341239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864105472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573774.341250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573774.341254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573774.341258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573774.341283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573774.341307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573774.341319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff92fc6733ad80 x1875449864105600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573774.341333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864105600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573774.341342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573774.341363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573774.341368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573774.341375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573774.341410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573774.341423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff92fc67339c00 x1875449864105728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573774.341436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864105728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573774.341444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573774.341448:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573774.341451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573774.341458:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573774.341480:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573774.341493:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573774/real 1788573774] req@ffff92fc67339880 x1875449864105856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573774.341513:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864105856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573774.341522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573774.341526:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573774.341529:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864105856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573774.788832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573774.852810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573775.812858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573775.876822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573776.836900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573776.900789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573777.860864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573777.924830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573778.884879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573778.948795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573779.460821:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.460832:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573779.460837:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.460857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5530 00000100:00080000:3.0:1788573779.460868:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.460878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573779.460912:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864105984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573779.460940:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.460945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573779.460946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.460951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5530 00000100:00080000:3.0:1788573779.460955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.460959:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573779.460968:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573779.460979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864106112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573779.460988:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.460991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573779.460993:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.460998:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5530 00000100:00080000:3.0:1788573779.461001:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573779.461006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864105984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573779.461006:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573779.461010:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573779.461021:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864106240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573779.461029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573779.461032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573779.461033:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.461036:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5530 00000100:00080000:3.0:1788573779.461039:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.461042:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573779.461046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573779.461079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573779.461112:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864106368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573779.461119:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573779.461124:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff92fc6733b100 x1875449864105984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573779.461124:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573779.461128:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573779.461134:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5530 00000100:00080000:3.0:1788573779.461140:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573779.461146:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573779.461151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573779.461158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864105984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573779.461161:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864106496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573779.461179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573779.461185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573779.461190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864105984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573779.461213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573779.461236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573779.461271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff92fc67338380 x1875449864106112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573779.461283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864106112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573779.461289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573779.461292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573779.461295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573779.461308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573779.461329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573779.461343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff92fc67338a80 x1875449864106240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573779.461360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864106240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573779.461370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573779.461375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573779.461379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573779.461386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573779.461406:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573779.461415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff92fc6733b800 x1875449864106368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573779.461425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864106368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573779.461433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573779.461436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573779.461440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573779.461457:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573779.461477:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573779.461486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573779/real 1788573779] req@ffff92fc67339500 x1875449864106496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573779.461494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864106496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573779.461499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573779.461501:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573779.461503:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573779.908853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573779.972821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573780.932895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573780.996821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573781.956855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573782.020920:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573782.980897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573783.044829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573784.004826:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573784.068825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573784.580899:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573784.580908:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573784.580911:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573784.580920:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5535 00000100:00080000:1.0:1788573784.580929:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573784.580935:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573784.580969:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864106624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573784.580995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573784.580998:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573784.580999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573784.581001:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5535 00000100:00080000:1.0:1788573784.581004:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573784.581007:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573784.581015:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573784.581024:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864106752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573784.581031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573784.581034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573784.581035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573784.581037:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5535 00000100:00080000:1.0:1788573784.581039:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573784.581043:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573784.581046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573784.581057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864106880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573784.581065:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864106624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573784.581069:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573784.581081:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573784.581083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573784.581087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5535 00000100:00080000:1.0:1788573784.581089:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573784.581092:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573784.581095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573784.581104:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864107008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573784.581111:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573784.581113:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573784.581115:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573784.581117:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5535 00000100:00080000:1.0:1788573784.581119:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573784.581122:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573784.581125:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573784.581132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864107136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573784.581184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573784.581252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff92fc67289180 x1875449864106624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573784.581276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864106624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573784.581292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573784.581296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573784.581299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864106624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573784.581306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573784.581328:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573784.581338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff92fc67289500 x1875449864106752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573784.581347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864106752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573784.581353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573784.581356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573784.581358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573784.581383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573784.581395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573784.581402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff92fc6728bb80 x1875449864106880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573784.581414:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864106880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573784.581420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573784.581422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573784.581424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864106880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573784.581428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573784.581439:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573784.581445:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff92fc67288700 x1875449864107008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573784.581453:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864107008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573784.581458:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573784.581459:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573784.581461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573784.581465:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573784.581475:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573784.581481:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff92fc67288a80 x1875449864107136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573784.581488:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864107136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573784.581498:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573784.581500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573784.581502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573785.028982:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573785.092858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573786.052852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573786.116875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573787.076892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573787.140826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573788.100824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573788.164791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573789.124862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573789.188782:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573789.700825:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.700835:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573789.700851:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.700861:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5540 00000100:00080000:3.0:1788573789.700871:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.700878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573789.700914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864107264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573789.700950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.700955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573789.700957:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.700962:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5540 00000100:00080000:3.0:1788573789.700966:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.700971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573789.700981:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573789.700983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864107264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573789.701019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864107392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573789.701032:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.701035:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573789.701038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.701042:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5540 00000100:00080000:3.0:1788573789.701047:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.701052:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573789.701057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573789.701076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573789.701083:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864107520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573789.701095:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573789.701098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573789.701100:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573789.701105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5540 00000100:00080000:3.0:1788573789.701110:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.701114:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573789.701120:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573789.701125:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff92fc67338700 x1875449864107264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573789.701136:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864107648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573789.701146:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864107264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573789.701171:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573789.701171:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573789.701179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788573789.701179:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573789.701185:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573789.701186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864107264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573789.701194:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5540 00000100:00080000:3.0:1788573789.701199:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573789.701203:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573789.701208:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573789.701217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573789.701221:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864107776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573789.701243:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573789.701255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff92fc6733bb80 x1875449864107392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573789.701269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864107392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573789.701276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573789.701278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573789.701280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573789.701284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573789.701295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573789.701301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff92fc6733b480 x1875449864107520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573789.701308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864107520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573789.701313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573789.701315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573789.701317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573789.701320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573789.701331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573789.701337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff92fc6733a300 x1875449864107648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573789.701344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864107648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573789.701353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573789.701355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573789.701357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573789.701372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573789.701391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573789.701402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff92fc67338000 x1875449864107776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573789.701415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864107776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573789.701422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573789.701424:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573789.701426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864107776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573790.148839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573790.212778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573791.172899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573791.236820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573792.196830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573792.260812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573793.220929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573793.284829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573794.244884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573794.308821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788573794.820846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573794.820858:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573794.820863:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573794.820874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5545 00000100:00080000:0.0:1788573794.820884:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573794.820890:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573794.820924:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864107904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573794.820955:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573794.820959:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573794.820961:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573794.820965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5545 00000100:00080000:0.0:1788573794.820970:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573794.820987:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573794.820999:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573794.821012:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864108032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573794.821021:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864107904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788573794.821030:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573794.821034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573794.821035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573794.821038:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5545 00000100:00080000:0.0:1788573794.821042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573794.821045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573794.821048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573794.821057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864108160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573794.821068:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573794.821070:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573794.821072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573794.821074:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5545 00000100:00080000:0.0:1788573794.821084:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573794.821087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573794.821090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573794.821098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864108288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573794.821103:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573794.821106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573794.821107:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573794.821110:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5545 00000100:00080000:0.0:1788573794.821112:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573794.821114:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788573794.821117:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573794.821126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864108416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573794.821143:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573794.821172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573794.821187:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573794.821200:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573795.268873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573795.332878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573796.292830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573796.356802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573797.316894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573797.380831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788573797.893110:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788573797.893165:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573797.893193:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573797.893246:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573797.893258:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573797.893268:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788573797.893280:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573797.893306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff92fc6704a300 x1875449864108032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.893363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864108032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.893382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.893389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573797.893396:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573797.893407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff92fc6704aa00 x1875449864108160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.893423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864108160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.893431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.893435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573797.893438:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573797.893446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff92fc6704a680 x1875449864108288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.893458:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864108288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.893466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.893470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573797.893473:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573797.893481:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff92fc6704ad80 x1875449864108416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.893493:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864108416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.893501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.893504:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573797.893507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108416:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573797.893515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573797] req@ffff92fc67049880 x1875449864107904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573797.893528:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864107904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573797.893536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573797.893539:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573797.893543:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864107904:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573798.340963:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573798.404852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573799.364876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573799.428832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573799.940831:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573799.940839:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573799.940843:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573799.940853:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5550 00000100:00080000:3.0:1788573799.940861:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573799.940867:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573799.940916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864108544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573799.940938:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573799.940941:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573799.940942:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573799.940946:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5550 00000100:00080000:3.0:1788573799.940951:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573799.940954:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573799.940966:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573799.940981:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864108672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573799.940992:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864108544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573799.940996:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573799.941002:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573799.941004:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573799.941010:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5550 00000100:00080000:3.0:1788573799.941014:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573799.941019:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573799.941025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573799.941039:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864108800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573799.941049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573799.941054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573799.941056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573799.941059:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5550 00000100:00080000:3.0:1788573799.941064:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573799.941069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573799.941074:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573799.941082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573799.941098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864108928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573799.941109:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573799.941114:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573799.941116:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff92fc6733a680 x1875449864108544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573799.941118:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573799.941124:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5550 00000100:00080000:3.0:1788573799.941129:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573799.941138:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573799.941147:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573799.941153:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864108544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573799.941163:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864109056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573799.941171:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.941180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573799.941186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864108544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573799.941218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573799.941242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573799.941255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff92fc67338e00 x1875449864108672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.941267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864108672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.941276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.941280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573799.941283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573799.941290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573799.941308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573799.941318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff92fc6733aa00 x1875449864108800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.941330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864108800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.941339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.941343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573799.941346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573799.941352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573799.941370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573799.941380:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff92fc67338000 x1875449864108928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.941397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864108928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.941405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.941408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573799.941412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864108928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573799.941418:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573799.941445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573799.941455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff92fc6733a300 x1875449864109056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573799.941466:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864109056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573799.941474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573799.941478:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573799.941482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573800.388864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573800.452780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573801.412878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573801.476836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573802.436899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573802.500834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573803.460901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573803.524819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573804.484886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573804.548837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573805.060856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573805.060871:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573805.060877:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573805.060890:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5555 00000100:00080000:3.0:1788573805.060904:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573805.060914:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573805.060966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864109184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573805.061003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573805.061008:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573805.061010:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573805.061014:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5555 00000100:00080000:3.0:1788573805.061019:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573805.061023:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573805.061036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573805.061054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864109312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573805.061058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864109184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573805.061094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573805.061098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573805.061102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573805.061108:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5555 00000100:00080000:3.0:1788573805.061112:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573805.061117:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573805.061125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573805.061138:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573805.061144:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864109440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573805.061160:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573805.061164:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573805.061167:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573805.061173:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5555 00000100:00080000:3.0:1788573805.061180:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573805.061184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573805/real 1788573805] req@ffff92fc6733b480 x1875449864109184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573805.061189:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573805.061207:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573805.061214:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864109184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573805.061224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573805.061226:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864109568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573805.061232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573805.061243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864109184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573805.061261:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573805.061266:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573805.061269:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573805.061271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109312:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573805.061280:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5555 00000100:00080000:3.0:1788573805.061288:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573805.061295:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788573805.061296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573805.061307:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573805.061310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573805/real 1788573805] req@ffff92fc6733bb80 x1875449864109312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573805.061325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864109312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573805.061333:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864109696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573805.061335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573805.061347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573805.061356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573805.061374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573805.061390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573805.061398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573805/real 1788573805] req@ffff92fc67338700 x1875449864109440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573805.061411:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864109440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573805.061420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573805.061423:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573805.061427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573805.061434:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573805.061456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573805.061468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573805/real 1788573805] req@ffff92fc67339500 x1875449864109568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573805.061478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864109568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573805.061483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573805.061485:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573805.061487:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573805.061500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573805.061515:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573805.061521:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573805/real 1788573805] req@ffff92fc6733b800 x1875449864109696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573805.061529:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864109696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573805.061534:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573805.061536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573805.061537:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573805.508921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573805.572840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573806.532919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573806.596885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573807.556922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573807.620850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573808.580953:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573808.644908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573809.604853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573809.668805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573810.180826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573810.180834:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573810.180837:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573810.180847:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5560 00000100:00080000:3.0:1788573810.180857:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573810.180865:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573810.180904:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864109824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573810.180932:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573810.180936:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573810.180939:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573810.180943:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5560 00000100:00080000:3.0:1788573810.180946:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573810.180950:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573810.180961:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573810.180975:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864109952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573810.180984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573810.180987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864109824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573810.180989:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573810.180993:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573810.181001:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5560 00000100:00080000:3.0:1788573810.181004:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573810.181014:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573810.181018:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573810.181031:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864110080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573810.181038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573810.181040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573810.181042:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573810.181044:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5560 00000100:00080000:3.0:1788573810.181046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573810.181048:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573810.181052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573810.181060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864110208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573810.181069:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573810.181071:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573810.181074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573810.181076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573810.181079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5560 00000100:00080000:3.0:1788573810.181083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573810.181085:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573810.181088:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573810.181095:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864110336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573810.181119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff92fc67338a80 x1875449864109824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.181142:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864109824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.181154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.181158:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573810.181163:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864109824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573810.181185:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573810.181205:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573810.181215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff92fc67338380 x1875449864109952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.181224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864109952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.181230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.181232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573810.181235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864109952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573810.181238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573810.181250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573810.181256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff92fc6733b100 x1875449864110080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.181270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864110080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.181278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.181282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573810.181285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573810.181291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573810.181312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573810.181325:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff92fc67339880 x1875449864110208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.181335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864110208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.181341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.181343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573810.181345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573810.181349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573810.181371:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573810.181379:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573810/real 1788573810] req@ffff92fc67339c00 x1875449864110336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573810.181386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864110336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573810.181396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573810.181398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573810.181401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573810.628835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573810.692804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573811.652948:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573811.716865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573812.676954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573812.740803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573813.700978:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573813.764866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573814.724926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573814.788849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573815.300851:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573815.300861:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573815.300865:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573815.300876:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5565 00000100:00080000:3.0:1788573815.300888:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573815.300894:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573815.300934:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864110464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573815.300963:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573815.300968:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573815.300970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573815.300975:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5565 00000100:00080000:3.0:1788573815.300979:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573815.300983:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573815.300994:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573815.301007:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864110592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573815.301015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573815.301018:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573815.301019:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573815.301022:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5565 00000100:00080000:3.0:1788573815.301026:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573815.301031:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573815.301036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573815.301051:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864110720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573815.301052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864110464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573815.301076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573815.301078:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573815.301079:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573815.301082:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5565 00000100:00080000:3.0:1788573815.301085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573815.301089:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573815.301093:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573815.301103:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864110848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573815.301110:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573815.301113:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573815.301113:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573815.301116:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5565 00000100:00080000:3.0:1788573815.301122:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573815.301126:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573815.301129:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573815.301137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864110976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573815.301153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573815.301215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff92fc6733ad80 x1875449864110464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.301254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864110464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.301268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.301272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573815.301275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864110464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573815.301283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573815.301316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573815.301332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff92fc67339f80 x1875449864110592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.301342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864110592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.301349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.301354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573815.301360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573815.301392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573815.301407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573815.301415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff92fc67339180 x1875449864110720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.301423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864110720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.301430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.301433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573815.301436:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573815.301443:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573815.301467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573815.301476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff92fc67339c00 x1875449864110848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.301485:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864110848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.301491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.301495:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573815.301498:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573815.301503:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573815.301517:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573815.301524:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573815/real 1788573815] req@ffff92fc67339880 x1875449864110976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573815.301542:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864110976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573815.301548:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573815.301550:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573815.301553:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864110976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573815.748885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573815.812799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573816.772843:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573816.836826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573817.796907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573817.860788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573818.820882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573818.884829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573819.844898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573819.908825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573820.420785:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573820.420794:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573820.420798:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573820.420806:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5571 00000100:00080000:3.0:1788573820.420815:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573820.420821:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573820.420852:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864111104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573820.420873:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573820.420876:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573820.420877:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573820.420880:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5571 00000100:00080000:3.0:1788573820.420882:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573820.420884:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573820.420893:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573820.420902:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864111232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573820.420909:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573820.420912:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573820.420914:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573820.420916:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5571 00000100:00080000:3.0:1788573820.420918:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573820.420923:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864111104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573820.420924:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573820.420934:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573820.420942:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864111360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573820.420950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573820.420952:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573820.420954:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573820.420957:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5571 00000100:00080000:3.0:1788573820.420959:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573820.420962:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573820.420966:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573820.420973:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864111488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573820.420979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573820.420982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573820.420983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573820.420986:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5571 00000100:00080000:3.0:1788573820.420988:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573820.420992:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573820.420993:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573820.420999:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573820.421007:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864111616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573820.421041:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff92fc6733b100 x1875449864111104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573820.421062:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864111104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573820.421073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573820.421077:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573820.421081:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864111104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573820.421089:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573820.421110:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573820.421123:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff92fc67338380 x1875449864111232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573820.421136:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864111232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573820.421142:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573820.421145:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573820.421147:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573820.421168:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573820.421181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573820.421189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff92fc67338a80 x1875449864111360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573820.421196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864111360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573820.421202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573820.421204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573820.421206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573820.421209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573820.421220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573820.421225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff92fc6733b800 x1875449864111488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573820.421232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864111488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573820.421238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573820.421240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573820.421243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573820.421246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573820.421255:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573820.421260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573820/real 1788573820] req@ffff92fc67339500 x1875449864111616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573820.421267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864111616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573820.421271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573820.421273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573820.421275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573820.868847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573820.932804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573821.892929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573821.956833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573822.916817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573822.980848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573823.940877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573824.004833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573824.964823:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573825.028813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573825.540847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573825.540854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573825.540858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573825.540867:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5576 00000100:00080000:2.0:1788573825.540876:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573825.540882:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573825.540914:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864111744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573825.540936:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573825.540939:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573825.540941:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573825.540943:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5576 00000100:00080000:2.0:1788573825.540945:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573825.540948:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573825.540955:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573825.540964:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864111872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573825.540971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573825.540974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573825.540975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573825.540978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5576 00000100:00080000:2.0:1788573825.540980:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573825.540985:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573825.540987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864111744:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788573825.540989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573825.540998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864112000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573825.541005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573825.541007:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573825.541008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573825.541010:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5576 00000100:00080000:2.0:1788573825.541013:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573825.541015:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573825.541018:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573825.541025:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864112128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573825.541031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573825.541033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573825.541034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573825.541036:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5576 00000100:00080000:2.0:1788573825.541039:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573825.541042:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573825.541045:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573825.541051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864112256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573825.541058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573825.541098:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff92fc4617dc00 x1875449864111744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.541112:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864111744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.541119:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.541123:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573825.541126:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864111744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573825.541132:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573825.541147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573825.541155:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff92fc4617ca80 x1875449864111872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.541163:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864111872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.541168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.541170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573825.541173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864111872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573825.541191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573825.541202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573825.541208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff92fc4617c700 x1875449864112000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.541215:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864112000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.541221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.541223:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573825.541225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573825.541228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573825.541239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573825.541245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff92fc4617f800 x1875449864112128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.541252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864112128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.541256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.541258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573825.541260:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573825.541263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573825.541272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573825.541278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573825/real 1788573825] req@ffff92fc4617f100 x1875449864112256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573825.541289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864112256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573825.541297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573825.541300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573825.541304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788573825.988943:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573826.052866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573827.012844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573827.076800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788573828.036888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573828.100848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573829.060905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573829.124808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573830.084902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573830.148858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573830.660805:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.660816:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573830.660821:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.660832:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5581 00000100:00080000:3.0:1788573830.660842:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.660850:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573830.660887:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864112384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573830.660915:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.660919:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573830.660922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.660926:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5581 00000100:00080000:3.0:1788573830.660931:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.660935:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573830.660945:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573830.660952:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864112384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573830.660978:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864112512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573830.660992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.660996:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573830.660999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.661002:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5581 00000100:00080000:3.0:1788573830.661006:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573830.661011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573830.661014:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573830.661020:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573830.661032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864112640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573830.661045:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.661048:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573830.661049:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573830.661052:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff92fc67338700 x1875449864112384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573830.661055:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5581 00000100:00080000:3.0:1788573830.661061:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.661067:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573830.661072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573830.661078:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864112384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573830.661083:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864112768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573830.661099:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573830.661100:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.661106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573830.661108:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788573830.661109:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573830.661116:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864112384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573830.661116:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5581 00000100:00080000:3.0:1788573830.661122:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.661126:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573830.661129:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573830.661135:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864112896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573830.661143:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573830.661161:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573830.661171:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff92fc6733bb80 x1875449864112512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.661179:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864112512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.661186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.661188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573830.661191:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573830.661194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573830.661210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573830.661221:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff92fc6733b480 x1875449864112640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.661233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864112640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.661248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.661252:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573830.661255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573830.661261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573830.661281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573830.661303:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff92fc6733a300 x1875449864112768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.661315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864112768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.661323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.661326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573830.661329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573830.661346:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573830.661381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573830.661393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff92fc67338000 x1875449864112896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.661404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864112896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.661412:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.661415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573830.661419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864112896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573831.108868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573831.172817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573832.132942:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573832.196821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573833.156919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573833.220844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573834.180975:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573834.244864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573835.204869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573835.268807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573835.780800:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573835.780812:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573835.780817:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573835.780829:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5586 00000100:00080000:3.0:1788573835.780841:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573835.780850:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573835.780887:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864113024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573835.780914:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573835.780918:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573835.780919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573835.780922:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5586 00000100:00080000:3.0:1788573835.780925:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573835.780928:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573835.780939:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573835.780954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864113152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573835.780967:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573835.780969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864113024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573835.780971:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573835.780976:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573835.780984:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5586 00000100:00080000:3.0:1788573835.780989:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573835.780994:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573835.781000:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573835.781016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864113280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573835.781028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573835.781032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573835.781034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573835.781041:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5586 00000800:00000100:1.0:1788573835.781043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573835.781046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573835.781051:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573835.781057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573835.781069:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864113408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573835.781076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573835.781078:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573835.781080:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573835.781082:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff92fc6733aa00 x1875449864113024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573835.781086:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5586 00000100:00080000:3.0:1788573835.781091:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573835.781106:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864113024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573835.781107:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573835.781115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573835.781119:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.781125:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573835.781128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864113536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573835.781133:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864113024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573835.781162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573835.781179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573835.781188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff92fc67338e00 x1875449864113152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.781196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864113152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.781202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.781204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573835.781206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573835.781210:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573835.781221:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573835.781228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff92fc6733a680 x1875449864113280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.781235:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864113280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.781241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.781242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573835.781244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573835.781248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573835.781259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573835.781284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff92fc67338000 x1875449864113408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.781292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864113408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.781298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.781300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573835.781302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573835.781305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573835.781318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573835.781324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff92fc6733a300 x1875449864113536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.781331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864113536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.781337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.781339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573835.781341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573836.228862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573836.292843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573837.252865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573837.316778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573838.276862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573838.340835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573839.300883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573839.364777:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573840.324832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573840.388820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573840.900838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573840.900848:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573840.900852:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573840.900863:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5591 00000100:00080000:3.0:1788573840.900873:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573840.900879:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573840.900921:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864113664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573840.900943:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573840.900946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573840.900947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573840.900950:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5591 00000100:00080000:3.0:1788573840.900952:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573840.900954:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573840.900963:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573840.900971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864113792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573840.900978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573840.900981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573840.900982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573840.900985:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5591 00000100:00080000:3.0:1788573840.900987:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573840.900989:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573840.900992:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573840.900998:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864113920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573840.901005:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864113664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573840.901007:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573840.901014:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573840.901017:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573840.901020:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5591 00000100:00080000:3.0:1788573840.901023:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573840.901025:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573840.901028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573840.901037:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864114048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573840.901044:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573840.901047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573840.901048:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573840.901051:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5591 00000100:00080000:3.0:1788573840.901053:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573840.901056:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573840.901058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573840.901065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864114176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573840.901077:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573840.901114:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff92fc6733b480 x1875449864113664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.901128:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864113664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.901136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.901139:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573840.901142:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864113664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573840.901147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573840.901167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573840.901217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff92fc6733bb80 x1875449864113792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.901231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864113792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.901240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.901244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573840.901248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573840.901274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573840.901297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573840.901306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff92fc67338700 x1875449864113920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.901315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864113920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.901320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.901323:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573840.901325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864113920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573840.901329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573840.901342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573840.901348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff92fc67339500 x1875449864114048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.901355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864114048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.901360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.901362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573840.901364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573840.901367:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573840.901378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573840.901384:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff92fc6733b800 x1875449864114176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573840.901391:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864114176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573840.901396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573840.901398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573840.901400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573841.348807:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573841.412853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573842.372832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573842.436828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573843.396851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573843.460828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573844.420863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573844.484832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573845.444913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573845.508890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573846.020787:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573846.020799:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573846.020804:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573846.020815:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5596 00000100:00080000:3.0:1788573846.020829:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573846.020839:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573846.020894:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864114304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573846.020922:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573846.020928:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573846.020929:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573846.020932:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5596 00000100:00080000:3.0:1788573846.020936:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573846.020938:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573846.020949:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573846.020960:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864114432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573846.020968:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573846.020971:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573846.020973:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573846.020975:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5596 00000100:00080000:3.0:1788573846.020977:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573846.020979:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573846.020982:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573846.020994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864114560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573846.020996:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864114304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573846.021008:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573846.021011:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573846.021012:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573846.021014:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5596 00000100:00080000:3.0:1788573846.021016:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573846.021018:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573846.021023:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573846.021032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864114688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573846.021037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573846.021040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573846.021041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573846.021044:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5596 00000100:00080000:3.0:1788573846.021046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573846.021049:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573846.021052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573846.021059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864114816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573846.021075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573846.021140:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573846/real 1788573846] req@ffff92fc67338a80 x1875449864114304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573846.021161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864114304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573846.021173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573846.021178:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573846.021183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864114304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573846.021191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573846.021216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573846.021228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573846/real 1788573846] req@ffff92fc67338380 x1875449864114432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573846.021252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864114432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573846.021270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573846.021275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573846.021278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573846.021307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573846.021324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573846.021332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573846/real 1788573846] req@ffff92fc6733b100 x1875449864114560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573846.021340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864114560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573846.021346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573846.021348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573846.021350:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573846.021353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573846.021364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573846.021370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573846/real 1788573846] req@ffff92fc67339880 x1875449864114688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573846.021377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864114688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573846.021382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573846.021383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573846.021386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573846.021390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573846.021401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573846.021406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573846/real 1788573846] req@ffff92fc67339c00 x1875449864114816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573846.021414:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864114816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573846.021419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573846.021421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573846.021423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864114816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573846.468832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573846.532818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573847.492844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573847.556832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573848.516824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573848.580853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573849.540868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573849.604833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573850.564875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573850.628831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573851.140830:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573851.140841:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573851.140846:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573851.140858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5601 00000100:00080000:3.0:1788573851.140869:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573851.140876:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573851.140924:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864114944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573851.140953:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573851.140958:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573851.140960:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573851.140964:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5601 00000100:00080000:3.0:1788573851.140969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573851.140973:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573851.140998:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573851.141015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864115072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573851.141021:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864114944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573851.141036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573851.141040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573851.141042:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573851.141056:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5601 00000100:00080000:3.0:1788573851.141061:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573851.141066:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573851.141071:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573851.141085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864115200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573851.141089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573851.141102:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573851.141106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573851.141109:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573851.141114:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5601 00000100:00080000:3.0:1788573851.141118:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573851.141122:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573851.141128:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573851.141130:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573851/real 1788573851] req@ffff92fc67339180 x1875449864114944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573851.141146:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864115328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573851.141150:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864114944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573851.141175:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573851.141182:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788573851.141186:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573851.141187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864114944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573851.141195:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573851.141200:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573851.141205:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5601 00000100:00080000:3.0:1788573851.141211:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573851.141214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115072:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573851.141219:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573851.141226:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573851.141237:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788573851.141242:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864115456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573851.141249:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573851/real 1788573851] req@ffff92fc67339f80 x1875449864115072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573851.141270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864115072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573851.141280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573851.141283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573851.141286:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573851.141291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573851.141307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573851.141315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573851/real 1788573851] req@ffff92fc6733ad80 x1875449864115200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573851.141325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864115200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573851.141330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573851.141334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573851.141345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573851.141353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573851.141370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573851.141377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573851/real 1788573851] req@ffff92fc67339c00 x1875449864115328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573851.141385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864115328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573851.141393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573851.141397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573851.141400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573851.141420:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573851.141448:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573851.141479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573851/real 1788573851] req@ffff92fc67339880 x1875449864115456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573851.141495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864115456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573851.141507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573851.141512:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573851.141516:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573851.588838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573851.652835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573852.612851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573852.676842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573853.636875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573853.700849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573854.660855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573854.724821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573855.684875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573855.748840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573856.260859:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573856.260867:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573856.260887:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573856.260896:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5606 00000100:00080000:1.0:1788573856.260906:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573856.260912:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573856.260958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864115584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573856.260990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573856.260994:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573856.260996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573856.260999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5606 00000100:00080000:1.0:1788573856.261002:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573856.261005:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573856.261013:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573856.261025:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864115712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573856.261033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573856.261035:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573856.261037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573856.261040:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5606 00000100:00080000:1.0:1788573856.261042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573856.261046:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573856.261048:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864115584:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788573856.261050:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573856.261059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864115840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573856.261066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573856.261068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573856.261069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573856.261071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5606 00000100:00080000:1.0:1788573856.261073:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573856.261075:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573856.261079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573856.261087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864115968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573856.261093:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573856.261096:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573856.261097:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573856.261099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5606 00000100:00080000:1.0:1788573856.261114:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573856.261116:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573856.261119:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573856.261128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864116096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573856.261129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573856.261178:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff92fc6728b100 x1875449864115584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573856.261210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864115584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573856.261222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573856.261226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573856.261231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864115584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573856.261244:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573856.261271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573856.261284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff92fc67288380 x1875449864115712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573856.261298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864115712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573856.261308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573856.261312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573856.261316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573856.261341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573856.261364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573856.261377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff92fc67289c00 x1875449864115840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573856.261390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864115840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573856.261400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573856.261403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573856.261407:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573856.261413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573856.261434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573856.261446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff92fc6728a680 x1875449864115968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573856.261459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864115968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573856.261468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573856.261471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573856.261475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864115968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573856.261481:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573856.261500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573856.261511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573856/real 1788573856] req@ffff92fc6728b800 x1875449864116096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573856.261523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864116096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573856.261532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573856.261535:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573856.261538:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573856.708917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573856.772791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573857.732959:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573857.796878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573858.756924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573858.820837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573859.780845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573859.844863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573860.804901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573860.868844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573861.380862:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573861.380870:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573861.380873:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573861.380882:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5611 00000100:00080000:3.0:1788573861.380891:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573861.380897:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573861.380931:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864116224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573861.380969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573861.380974:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573861.380977:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573861.380982:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5611 00000100:00080000:3.0:1788573861.380986:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573861.380991:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573861.381003:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573861.381008:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864116224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573861.381019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864116352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573861.381033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573861.381037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573861.381039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573861.381043:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5611 00000100:00080000:3.0:1788573861.381046:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573861.381050:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573861.381055:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573861.381065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864116480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573861.381073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573861.381076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573861.381077:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573861.381080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5611 00000100:00080000:3.0:1788573861.381083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573861.381086:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573861.381090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573861.381099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864116608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573861.381106:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573861.381109:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573861.381111:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573861.381115:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5611 00000100:00080000:3.0:1788573861.381117:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573861.381120:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573861.381124:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573861.381129:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573861.381134:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864116736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573861.381169:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573861.381191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573861.381217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573861.828805:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573861.892835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573862.852924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573862.916911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573863.876815:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573863.940829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788573864.453045:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788573864.453093:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573864.453110:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573864.453160:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573864.453170:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573864.453178:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573864.453187:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788573864.453968:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff92fc67339500 x1875449864116736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573864.453998:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864116736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573864.454010:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573864.454016:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573864.454020:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116736:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573864.454028:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff92fc67338380 x1875449864116352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573864.454041:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864116352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573864.454049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573864.454052:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573864.454055:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573864.454062:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff92fc67338a80 x1875449864116480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573864.454072:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864116480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573864.454079:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573864.454083:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573864.454086:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573864.454094:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff92fc6733b800 x1875449864116608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573864.454106:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864116608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573864.454113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573864.454116:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573864.454119:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573864.454126:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573861/real 1788573864] req@ffff92fc6733b100 x1875449864116224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573864.454139:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864116224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573864.454147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573864.454151:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573864.454154:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864116224:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573864.900887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573864.964838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573865.924810:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573865.988830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573866.500821:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573866.500829:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573866.500832:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573866.500841:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5617 00000100:00080000:3.0:1788573866.500850:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573866.500856:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573866.500887:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864116864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573866.500909:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573866.500912:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573866.500913:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573866.500917:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5617 00000100:00080000:3.0:1788573866.500921:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573866.500926:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573866.500936:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573866.500950:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864116992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573866.500958:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864116864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573866.500977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573866.500981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573866.500984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573866.500989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5617 00000100:00080000:3.0:1788573866.500993:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573866.500997:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573866.501003:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573866.501024:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864117120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573866.501035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573866.501038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573866.501043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573866.501046:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573866.501051:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5617 00000100:00080000:3.0:1788573866.501056:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573866.501061:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573866.501067:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573866.501081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864117248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573866.501085:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff92fc67338700 x1875449864116864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573866.501101:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573866.501109:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573866.501112:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573866.501123:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5617 00000100:00080000:3.0:1788573866.501128:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573866.501133:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573866.501138:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864116864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573866.501140:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573866.501156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573866.501163:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573866.501167:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864117376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573866.501168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864116864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573866.501211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573866.501239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573866.501252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff92fc6733bb80 x1875449864116992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573866.501263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864116992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573866.501273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573866.501276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573866.501279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864116992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573866.501285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573866.501303:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573866.501313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff92fc6733b480 x1875449864117120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573866.501325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864117120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573866.501339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573866.501342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573866.501345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573866.501351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573866.501371:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573866.501382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff92fc6733a300 x1875449864117248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573866.501393:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864117248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573866.501401:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573866.501404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573866.501407:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573866.501413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573866.501430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573866.501442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573866/real 1788573866] req@ffff92fc67338000 x1875449864117376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573866.501452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864117376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573866.501461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573866.501463:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573866.501467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573866.948843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573867.012838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573867.972867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573868.036829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573868.996878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573869.060837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573870.020891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573870.084815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573871.044913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573871.108848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573871.620846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.620859:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573871.620863:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.620872:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5622 00000100:00080000:3.0:1788573871.620882:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.620889:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573871.620923:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864117504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.620945:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.620948:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.620950:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.620952:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5622 00000100:00080000:3.0:1788573871.620955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.620957:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573871.620964:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.620973:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864117632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.620982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.620984:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573871.620986:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.620990:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5622 00000100:00100000:0.0:1788573871.620991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864117504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573871.620993:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.620999:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573871.621002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.621010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864117760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.621015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.621017:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.621019:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.621021:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5622 00000100:00080000:3.0:1788573871.621022:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.621024:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573871.621027:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573871.621035:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864117888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.621041:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.621043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573871.621044:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.621046:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5622 00000100:00080000:3.0:1788573871.621049:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.621052:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573871.621055:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573871.621061:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864118016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573871.621081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573871.621117:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff92fc6733a680 x1875449864117504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573871.621131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864117504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573871.621140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573871.621143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573871.621146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864117504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573871.621173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573871.621197:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573871.621211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff92fc67338e00 x1875449864117632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573871.621224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864117632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573871.621233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573871.621242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573871.621245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573871.621249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573871.621265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573871.621274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff92fc6733aa00 x1875449864117760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573871.621282:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864117760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573871.621287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573871.621289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573871.621292:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573871.621295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573871.621306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573871.621312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff92fc67338000 x1875449864117888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573871.621319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864117888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573871.621324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573871.621326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573871.621328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864117888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573871.621331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573871.621342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573871.621347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff92fc6733a300 x1875449864118016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573871.621355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864118016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573871.621360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573871.621361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573871.621363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573872.068943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573872.132840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573873.092904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573873.156852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573874.116849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573874.180805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573875.140848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573875.204822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573876.164939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573876.228785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573876.740886:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.740898:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573876.740902:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.740913:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5627 00000100:00080000:2.0:1788573876.740925:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.740934:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573876.740984:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864118144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.741021:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.741028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573876.741030:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.741035:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5627 00000100:00080000:2.0:1788573876.741040:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.741045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573876.741058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573876.741079:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864118144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573876.741079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864118272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.741099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.741103:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573876.741106:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.741110:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5627 00000100:00080000:2.0:1788573876.741115:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.741118:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573876.741123:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573876.741143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864118400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.741151:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.741153:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573876.741155:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573876.741160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788573876.741164:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5627 00000100:00080000:2.0:1788573876.741168:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.741171:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573876.741176:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573876.741185:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864118528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573876.741192:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573876.741194:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573876.741195:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573876.741197:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5627 00000100:00080000:2.0:1788573876.741199:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573876.741202:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573876.741204:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573876.741213:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864118656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573876.741225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff92fc4617ed80 x1875449864118144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573876.741247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864118144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573876.741257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573876.741260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573876.741264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864118144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573876.741291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573876.741311:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573876.741320:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff92fc4617f480 x1875449864118272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573876.741329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864118272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573876.741335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573876.741337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573876.741340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573876.741343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573876.741354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573876.741361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff92fc4617f100 x1875449864118400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573876.741368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864118400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573876.741374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573876.741375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573876.741377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573876.741381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573876.741393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573876.741398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff92fc4617f800 x1875449864118528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573876.741411:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864118528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573876.741416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573876.741418:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573876.741426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573876.741429:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573876.741441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573876.741447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573876] req@ffff92fc4617c700 x1875449864118656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573876.741455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864118656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573876.741487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573876.741490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573876.741492:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573877.188892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573877.252800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573878.212851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573878.276807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573879.236825:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573879.300814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573880.260852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573880.324802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573881.284829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573881.348772:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573881.860836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573881.860847:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573881.860851:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573881.860862:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5632 00000100:00080000:2.0:1788573881.860873:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573881.860880:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573881.860917:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864118784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573881.860948:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573881.860952:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573881.860954:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573881.860959:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5632 00000100:00080000:2.0:1788573881.860964:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573881.860968:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573881.860978:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573881.860992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864118912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573881.861004:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573881.861010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864118784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573881.861010:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573881.861014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573881.861023:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5632 00000100:00080000:2.0:1788573881.861027:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573881.861030:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573881.861035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573881.861048:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864119040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573881.861058:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573881.861061:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573881.861063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573881.861067:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5632 00000100:00080000:2.0:1788573881.861071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573881.861075:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573881.861080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573881.861087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573881.861093:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864119168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573881.861120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573881.861124:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573881.861126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573881.861130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5632 00000100:00100000:0.0:1788573881.861133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff92fc4617ca80 x1875449864118784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573881.861136:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573881.861144:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573881.861151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573881.861155:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864118784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573881.861165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788573881.861167:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864119296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573881.861170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573881.861178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864118784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573881.861205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573881.861223:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573881.861231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff92fc4617dc00 x1875449864118912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573881.861239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864118912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573881.861245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573881.861247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573881.861250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864118912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573881.861253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573881.861263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573881.861269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff92fc4617e680 x1875449864119040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573881.861277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864119040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573881.861293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573881.861296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573881.861300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573881.861306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573881.861327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573881.861338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff92fc4617d180 x1875449864119168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573881.861347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864119168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573881.861352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573881.861379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573881.861382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573881.861386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573881.861402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573881.861409:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff92fc4617c380 x1875449864119296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573881.861417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864119296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573881.861422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573881.861424:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573881.861426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573882.308912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573882.372812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573883.332825:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573883.396817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573884.356918:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573884.420839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573885.380884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573885.444878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573886.404854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573886.468849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573886.980912:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.980921:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573886.980935:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.980952:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5637 00000100:00080000:3.0:1788573886.980963:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.980972:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573886.981010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864119424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573886.981038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.981043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573886.981046:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.981052:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5637 00000100:00080000:3.0:1788573886.981057:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.981063:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573886.981073:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864119424:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573886.981073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573886.981090:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864119552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573886.981099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.981102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573886.981104:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.981107:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5637 00000100:00080000:3.0:1788573886.981109:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.981112:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573886.981116:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573886.981125:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864119680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573886.981132:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.981135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788573886.981148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573886.981179:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff92fc6733b480 x1875449864119424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573886.981194:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864119424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573886.981203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573886.981204:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573886.981208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573886.981215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864119424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573886.981217:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5637 00000100:00080000:3.0:1788573886.981225:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.981231:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573886.981237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119552:192.168.202.132@tcp:38:kworker.0 00000008:00100000:3.0:1788573886.981238:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573886.981276:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864119808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573886.981283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573886.981297:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.981301:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573886.981303:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573886.981306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff92fc6733bb80 x1875449864119552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.981312:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5637 00000100:00080000:3.0:1788573886.981321:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573886.981322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864119552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573886.981330:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788573886.981348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573886.981353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573886.981358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573886.981365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573886.981388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573886.981399:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff92fc67338700 x1875449864119680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573886.981407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864119680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573886.981413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573886.981415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573886.981417:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573886.981427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573886.981442:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573886.981451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff92fc6733b100 x1875449864119808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573886.981459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864119808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573886.981465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573886.981467:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573886.981470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119808:192.168.202.132@tcp:8:kworker.0 00000008:00100000:3.0:1788573886.981479:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573886.981497:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864119936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573886.981597:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573886.981620:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573886.981631:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff92fc6733b800 x1875449864119936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573886.981640:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864119936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573886.981645:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573886.981647:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573886.981650:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864119936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573887.428840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573887.492811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573888.452911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573888.516912:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573889.476845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573889.540809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573890.500879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573890.564865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573891.524883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573891.588807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573892.100838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573892.100846:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573892.100849:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573892.100858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5642 00000100:00080000:1.0:1788573892.100866:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573892.100873:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573892.100903:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864120064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573892.100923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573892.100926:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573892.100927:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573892.100930:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5642 00000100:00080000:1.0:1788573892.100932:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573892.100934:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573892.100942:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573892.100949:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864120192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573892.100956:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573892.100958:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573892.100959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573892.100962:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5642 00000100:00080000:1.0:1788573892.100964:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573892.100967:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573892.100969:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573892.100975:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864120320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573892.100981:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573892.100986:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788573892.100987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864120064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573892.100989:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573892.100996:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5642 00000100:00080000:1.0:1788573892.100999:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573892.101002:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573892.101005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573892.101011:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864120448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573892.101017:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573892.101020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573892.101021:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573892.101024:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5642 00000100:00080000:1.0:1788573892.101026:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573892.101028:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573892.101031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573892.101038:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864120576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573892.101058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573892.101100:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573892/real 1788573892] req@ffff92fc67288e00 x1875449864120064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573892.101117:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864120064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573892.101125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573892.101129:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573892.101131:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864120064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573892.101137:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573892.101181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573892.101191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573892/real 1788573892] req@ffff92fc67289f80 x1875449864120192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573892.101200:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864120192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573892.101207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573892.101209:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573892.101211:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573892.101242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573892.101263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573892.101275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573892/real 1788573892] req@ffff92fc67288000 x1875449864120320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573892.101290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864120320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573892.101298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573892.101300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573892.101303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573892.101307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573892.101322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573892.101330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573892/real 1788573892] req@ffff92fc67289880 x1875449864120448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573892.101338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864120448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573892.101344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573892.101346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573892.101348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573892.101351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573892.101361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573892.101367:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573892/real 1788573892] req@ffff92fc6728ad80 x1875449864120576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573892.101374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864120576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573892.101379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573892.101381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573892.101382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573892.548845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573892.612830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573893.572849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573893.636787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573894.596914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573894.660870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573895.620857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573895.684799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573896.644844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573896.708852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573897.220794:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573897.220802:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573897.220806:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573897.220815:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5647 00000100:00080000:3.0:1788573897.220823:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573897.220829:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573897.220862:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864120704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573897.220885:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573897.220888:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573897.220890:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573897.220893:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5647 00000100:00080000:3.0:1788573897.220895:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573897.220897:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573897.220905:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573897.220914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864120832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573897.220922:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573897.220925:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573897.220927:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573897.220930:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5647 00000100:00080000:3.0:1788573897.220932:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573897.220935:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573897.220938:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573897.220947:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864120960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573897.220950:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864120704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573897.220966:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573897.220968:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573897.220969:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573897.220971:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5647 00000100:00080000:3.0:1788573897.220973:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573897.220976:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573897.220979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573897.220986:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864121088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573897.220992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573897.220994:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573897.220995:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573897.220997:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5647 00000100:00080000:3.0:1788573897.220999:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573897.221002:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573897.221007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573897.221018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864121216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573897.221038:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573897.221100:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff92fc67338a80 x1875449864120704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573897.221121:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864120704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573897.221133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573897.221138:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573897.221143:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864120704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573897.221151:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573897.221175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573897.221200:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff92fc67338380 x1875449864120832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573897.221214:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864120832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573897.221224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573897.221228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573897.221232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573897.221256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573897.221278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573897.221290:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff92fc67339500 x1875449864120960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573897.221303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864120960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573897.221312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573897.221315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573897.221319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864120960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573897.221326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573897.221345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573897.221356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff92fc67339880 x1875449864121088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573897.221368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864121088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573897.221377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573897.221381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573897.221384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573897.221390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573897.221410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573897.221420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573897/real 1788573897] req@ffff92fc67339c00 x1875449864121216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573897.221433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864121216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573897.221441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573897.221445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573897.221448:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573897.668851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573897.732821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573898.692821:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573898.756834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573899.716900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573899.780895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573900.740842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573900.804842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573901.764929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573901.828860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573902.340884:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573902.340897:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573902.340902:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573902.340913:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5652 00000100:00080000:1.0:1788573902.340926:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573902.340933:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573902.341001:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864121344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573902.341032:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573902.341036:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573902.341037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573902.341040:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5652 00000100:00080000:1.0:1788573902.341043:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573902.341046:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573902.341056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573902.341066:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864121472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573902.341073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573902.341076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573902.341077:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573902.341080:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5652 00000100:00080000:1.0:1788573902.341082:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573902.341084:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573902.341087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573902.341096:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864121344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573902.341096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864121600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573902.341112:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573902.341114:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573902.341115:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573902.341118:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5652 00000100:00080000:1.0:1788573902.341121:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573902.341123:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573902.341126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573902.341137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864121728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573902.341143:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573902.341145:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573902.341146:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573902.341148:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5652 00000100:00080000:1.0:1788573902.341151:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573902.341153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573902.341157:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573902.341165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864121856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573902.341191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573902.341257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff92fc6728a300 x1875449864121344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573902.341279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864121344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573902.341292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573902.341298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573902.341302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864121344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573902.341311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573902.341337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573902.341351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff92fc6728b480 x1875449864121472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573902.341366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864121472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573902.341376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573902.341380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573902.341384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573902.341417:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573902.341439:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573902.341452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff92fc6728aa00 x1875449864121600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573902.341466:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864121600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573902.341475:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573902.341479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573902.341483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573902.341490:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573902.341513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573902.341525:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff92fc6728ad80 x1875449864121728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573902.341539:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864121728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573902.341548:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573902.341552:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573902.341556:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573902.341563:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573902.341584:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573902.341597:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573902/real 1788573902] req@ffff92fc67289880 x1875449864121856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573902.341611:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864121856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573902.341620:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573902.341624:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573902.341627:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864121856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573902.788841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573902.852838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573903.812899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573903.876823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573904.836838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573904.900865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573905.860846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573905.924857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573906.884918:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573906.948821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573907.460832:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573907.460869:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573907.460874:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573907.460885:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5658 00000100:00080000:1.0:1788573907.460896:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573907.460904:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573907.460952:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864121984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573907.460979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573907.460985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573907.460987:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573907.460992:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5658 00000100:00080000:1.0:1788573907.460996:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573907.461000:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573907.461011:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573907.461024:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864121984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788573907.461026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864122112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573907.461041:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573907.461044:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573907.461046:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573907.461049:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5658 00000100:00080000:1.0:1788573907.461053:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573907.461057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573907.461062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573907.461073:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864122240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573907.461081:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573907.461087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788573907.461088:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573907.461092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573907.461097:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5658 00000100:00080000:1.0:1788573907.461102:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573907.461106:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573907.461113:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573907.461125:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864122368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573907.461128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff92fc67288000 x1875449864121984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573907.461153:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573907.461157:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788573907.461159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864121984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573907.461160:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573907.461171:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5658 00000100:00080000:0.0:1788573907.461177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573907.461177:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573907.461184:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788573907.461185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573907.461194:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864121984:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788573907.461195:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573907.461210:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864122496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573907.461228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573907.461257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573907.461268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff92fc67289f80 x1875449864122112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573907.461277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864122112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573907.461283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573907.461285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573907.461288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573907.461292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573907.461305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573907.461311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff92fc67288e00 x1875449864122240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573907.461319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864122240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573907.461324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573907.461326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573907.461336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573907.461340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573907.461353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573907.461360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff92fc6728b800 x1875449864122368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573907.461368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864122368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573907.461374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573907.461375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573907.461378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573907.461381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573907.461391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573907.461397:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573907/real 1788573907] req@ffff92fc6728a680 x1875449864122496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573907.461404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864122496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573907.461414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573907.461416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573907.461418:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573907.908911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573907.972869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573908.932864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573908.996858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573909.956867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573910.020896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573910.980934:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573911.044804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573912.004887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573912.068825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573912.580895:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.580907:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573912.580912:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.580924:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5663 00000100:00080000:3.0:1788573912.580935:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.580944:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573912.580981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864122624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573912.581009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.581013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.581015:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.581019:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5663 00000100:00080000:3.0:1788573912.581023:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.581028:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573912.581038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573912.581051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864122752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573912.581060:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864122624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573912.581066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.581071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573912.581073:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.581079:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5663 00000100:00080000:3.0:1788573912.581083:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.581087:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573912.581094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573912.581109:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864122880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573912.581121:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573912.581121:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.581128:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.581130:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.581136:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5663 00000100:00080000:3.0:1788573912.581141:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.581145:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573912.581152:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573912.581167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864123008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573912.581170:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff92fc6733ad80 x1875449864122624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573912.581194:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.581199:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573912.581201:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573912.581207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864122624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573912.581209:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5663 00000100:00080000:3.0:1788573912.581217:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573912.581223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573912.581225:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788573912.581229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573912.581234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864122624:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573912.581235:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573912.581251:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864123136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573912.581258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573912.581284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573912.581297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff92fc67339f80 x1875449864122752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573912.581306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864122752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573912.581312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573912.581315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573912.581317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573912.581321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573912.581332:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573912.581338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff92fc67339180 x1875449864122880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573912.581346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864122880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573912.581352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573912.581354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573912.581356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864122880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573912.581359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573912.581370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573912.581376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff92fc67339c00 x1875449864123008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573912.581384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864123008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573912.581389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573912.581390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573912.581392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573912.581401:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573912.581412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573912.581418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff92fc67339880 x1875449864123136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573912.581429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864123136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573912.581444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573912.581447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573912.581450:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573913.028983:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573913.092868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573914.053002:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573914.116910:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573915.076919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573915.140792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573916.100842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573916.164869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573917.124873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573917.188867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573917.700816:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.700824:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573917.700827:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.700837:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5668 00000100:00080000:3.0:1788573917.700845:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.700859:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573917.700898:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864123264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573917.700942:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.700946:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573917.700947:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.700951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5668 00000100:00080000:3.0:1788573917.700954:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.700957:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573917.700966:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573917.700977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864123392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573917.700985:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.700988:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788573917.700993:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864123264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573917.700998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.701003:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5668 00000100:00080000:3.0:1788573917.701006:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.701009:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573917.701013:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573917.701022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864123520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573917.701029:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.701032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573917.701033:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.701036:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5668 00000100:00080000:3.0:1788573917.701039:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.701041:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573917.701044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573917.701053:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864123648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573917.701064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573917.701068:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.701072:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573917.701073:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.701077:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5668 00000100:00080000:3.0:1788573917.701080:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.701083:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573917.701086:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573917.701093:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864123776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573917.701125:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff92fc67339500 x1875449864123264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573917.701143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864123264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573917.701153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573917.701156:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573917.701159:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864123264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573917.701177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573917.701196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573917.701205:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff92fc67338380 x1875449864123392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573917.701221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864123392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573917.701228:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573917.701230:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573917.701232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573917.701236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573917.701248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573917.701255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff92fc67338a80 x1875449864123520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573917.701263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864123520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573917.701268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573917.701270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573917.701272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573917.701275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573917.701287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573917.701293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff92fc6733b800 x1875449864123648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573917.701300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864123648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573917.701306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573917.701308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573917.701310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573917.701313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573917.701323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573917.701328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff92fc6733b100 x1875449864123776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573917.701335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864123776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573917.701340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573917.701342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573917.701344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864123776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573918.148876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573918.212821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573919.172860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573919.236795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573920.196809:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573920.260800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573921.221011:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573921.284825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573922.244868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573922.308884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573922.820938:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.820946:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573922.820951:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.820975:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5673 00000100:00080000:3.0:1788573922.820989:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.820998:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573922.821037:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864123904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.821067:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.821073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573922.821074:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.821077:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5673 00000100:00080000:3.0:1788573922.821081:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.821085:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573922.821095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573922.821108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864124032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.821117:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.821120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573922.821124:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573922.821125:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864123904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573922.821131:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5673 00000100:00080000:3.0:1788573922.821136:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.821138:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573922.821142:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573922.821161:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864124160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.821168:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.821170:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573922.821172:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.821176:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5673 00000100:00080000:3.0:1788573922.821179:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573922.821182:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573922.821188:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573922.821196:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864124288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573922.821202:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573922.821205:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573922.821206:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573922.821210:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5673 00000100:00080000:3.0:1788573922.821213:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573922.821214:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573922.821260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff92fc67338700 x1875449864123904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.821279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864123904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573922.821288:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:1.0:1788573922.821289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.821296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:3.0:1788573922.821298:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573922.821300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864123904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573922.821319:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864124416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573922.821329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573922.821354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573922.821365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff92fc6733bb80 x1875449864124032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.821378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864124032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.821385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.821388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573922.821391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573922.821396:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788573922.821410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573922.821419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff92fc6733b480 x1875449864124160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.821429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864124160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.821440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.821445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573922.821449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573922.821457:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573922.821489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573922.821504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff92fc6733a300 x1875449864124288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.821526:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864124288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.821536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.821541:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573922.821544:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573922.821565:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788573922.821592:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573922.821605:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff92fc67338000 x1875449864124416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.821620:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864124416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.821631:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.821635:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573922.821639:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573923.268839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573923.332831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573924.292844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573924.356868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573925.316859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573925.380901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573926.340878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573926.404851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573927.364868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573927.428852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573927.940837:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.940845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573927.940849:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.940859:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5678 00000100:00080000:3.0:1788573927.940871:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.940879:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573927.940918:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864124544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.940943:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.940947:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573927.940948:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.940951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5678 00000100:00080000:3.0:1788573927.940954:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.940957:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573927.940966:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573927.940976:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864124672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.940985:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573927.940986:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864124544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573927.940990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573927.940994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.940998:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5678 00000100:00080000:3.0:1788573927.941000:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.941002:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573927.941005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573927.941041:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864124800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.941048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.941051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573927.941052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.941055:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5678 00000100:00080000:3.0:1788573927.941058:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573927.941061:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573927.941065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573927.941077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864124928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.941093:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573927.941097:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573927.941099:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573927.941104:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5678 00000100:00080000:3.0:1788573927.941109:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573927.941114:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124672:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573927.941116:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573927.941125:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573927.941138:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864125056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573927.941145:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573927.941162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573927.941179:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573928.388868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573928.452799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573929.412869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573929.476844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573930.436852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573930.500853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788573931.013155:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788573931.013214:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573931.013233:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573931.013295:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573931.013308:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573931.013317:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788573931.013325:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788573931.013532:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573931] req@ffff92fc6733a300 x1875449864125056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573931.013566:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864125056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573931.013579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573931.013584:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573931.013589:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125056:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573931.013597:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573931] req@ffff92fc67338e00 x1875449864124672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573931.013610:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864124672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573931.013619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573931.013623:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573931.013626:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573931.013633:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573931] req@ffff92fc6733a680 x1875449864124800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573931.013645:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864124800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573931.013652:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573931.013655:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573931.013658:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788573931.013666:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573931] req@ffff92fc67338000 x1875449864124928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573931.013678:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864124928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573931.013696:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573931.013705:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573931.013708:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864124928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788573931.013715:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573931] req@ffff92fc6733aa00 x1875449864124544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573931.013774:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864124544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573931.013783:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573931.013787:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788573931.013796:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864124544:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788573931.460879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573931.524853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573932.484918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573932.548891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788573933.060872:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573933.060884:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573933.060888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573933.060898:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5683 00000100:00080000:1.0:1788573933.060911:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573933.060919:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788573933.060963:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864125184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573933.061011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573933.061016:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573933.061018:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573933.061023:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5683 00000100:00080000:1.0:1788573933.061027:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573933.061032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573933.061046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573933.061069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864125312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573933.061092:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573933.061099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864125184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788573933.061099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573933.061102:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573933.061112:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5683 00000100:00080000:1.0:1788573933.061118:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573933.061123:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573933.061143:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573933.061159:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864125440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573933.061171:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573933.061176:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573933.061178:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573933.061183:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5683 00000100:00080000:1.0:1788573933.061188:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573933.061193:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573933.061199:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573933.061208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788573933.061216:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864125568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573933.061227:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573933.061231:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573933.061233:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573933.061238:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5683 00000100:00080000:1.0:1788573933.061242:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573933.061246:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788573933.061252:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573933.061265:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864125696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573933.061269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573933/real 1788573933] req@ffff92fc67289c00 x1875449864125184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573933.061309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864125184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573933.061322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573933.061327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573933.061331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864125184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573933.061365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573933.061391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573933.061405:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573933/real 1788573933] req@ffff92fc67288380 x1875449864125312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573933.061418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864125312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573933.061427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573933.061430:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573933.061433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573933.061439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573933.061460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573933.061472:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573933/real 1788573933] req@ffff92fc6728b100 x1875449864125440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573933.061484:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864125440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573933.061499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573933.061503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573933.061507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573933.061514:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573933.061536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573933.061548:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573933/real 1788573933] req@ffff92fc67288a80 x1875449864125568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573933.061560:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864125568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573933.061570:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573933.061573:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573933.061577:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573933.061584:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573933.061607:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573933.061620:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573933/real 1788573933] req@ffff92fc67288700 x1875449864125696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573933.061632:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864125696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573933.061640:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573933.061643:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573933.061647:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573933.508893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573933.572872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573934.532921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573934.596793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573935.556909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573935.620884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573936.580854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573936.644826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573937.604838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573937.668833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573938.180854:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573938.180862:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573938.180866:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573938.180874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5688 00000100:00080000:3.0:1788573938.180884:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573938.180891:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573938.180931:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864125824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573938.180957:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573938.180960:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573938.180961:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573938.180964:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5688 00000100:00080000:3.0:1788573938.180967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573938.180970:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573938.180979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573938.180990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864125952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573938.180998:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573938.181000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573938.181003:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573938.181004:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864125824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573938.181007:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5688 00000100:00080000:3.0:1788573938.181012:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573938.181015:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573938.181019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573938.181028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864126080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573938.181034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573938.181037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573938.181038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573938.181040:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5688 00000100:00080000:3.0:1788573938.181042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573938.181044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573938.181048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573938.181054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864126208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573938.181060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573938.181062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573938.181064:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573938.181066:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5688 00000800:00000100:0.0:1788573938.181070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573938.181071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573938.181077:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573938.181080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573938.181087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864126336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573938.181127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff92fc6733b480 x1875449864125824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573938.181157:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864125824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573938.181166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573938.181170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573938.181172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864125824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573938.181195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573938.181230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573938.181243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff92fc6733bb80 x1875449864125952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573938.181253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864125952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573938.181259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573938.181262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573938.181264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864125952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573938.181268:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573938.181283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573938.181289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff92fc67338700 x1875449864126080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573938.181297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864126080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573938.181303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573938.181305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573938.181307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573938.181310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573938.181322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573938.181327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff92fc6733b100 x1875449864126208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573938.181335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864126208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573938.181340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573938.181341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573938.181344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573938.181347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573938.181357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573938.181363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573938/real 1788573938] req@ffff92fc6733b800 x1875449864126336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573938.181370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864126336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573938.181375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573938.181377:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573938.181379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573938.628870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573938.692849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573939.652869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573939.716814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573940.676881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573940.740830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573941.700826:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573941.764821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573942.724873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573942.788805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573943.300801:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573943.300811:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573943.300815:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573943.300826:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5693 00000100:00080000:3.0:1788573943.300837:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573943.300845:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573943.300881:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864126464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573943.300906:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573943.300909:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573943.300911:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573943.300913:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5693 00000100:00080000:3.0:1788573943.300916:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573943.300919:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573943.300926:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573943.300935:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864126592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573943.300947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573943.300949:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573943.300950:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573943.300953:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5693 00000100:00080000:3.0:1788573943.300956:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573943.300960:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573943.300965:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573943.300977:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864126720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573943.300983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864126464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573943.300995:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573943.300999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573943.301001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573943.301007:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5693 00000100:00080000:3.0:1788573943.301010:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573943.301013:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573943.301018:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573943.301028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864126848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573943.301043:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573943.301047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573943.301048:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573943.301052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573943.301056:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5693 00000100:00080000:3.0:1788573943.301060:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573943.301063:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573943.301067:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573943.301075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864126976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573943.301097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff92fc67338a80 x1875449864126464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573943.301118:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864126464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573943.301127:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573943.301130:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573943.301134:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864126464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573943.301139:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573943.301153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573943.301161:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff92fc67338380 x1875449864126592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573943.301169:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864126592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573943.301174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573943.301176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573943.301178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573943.301195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573943.301222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573943.301231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff92fc67339500 x1875449864126720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573943.301239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864126720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573943.301248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573943.301250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573943.301253:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573943.301256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573943.301267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573943.301273:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff92fc67339880 x1875449864126848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573943.301280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864126848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573943.301285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573943.301286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573943.301288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573943.301291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573943.301301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573943.301306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573943/real 1788573943] req@ffff92fc67339c00 x1875449864126976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573943.301313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864126976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573943.301318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573943.301319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573943.301322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864126976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573943.748842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573943.812822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573944.772865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573944.836789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573945.796817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573945.860834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573946.820874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573946.884828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573947.844860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573947.908799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573948.420839:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573948.420851:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573948.420855:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573948.420867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5699 00000100:00080000:3.0:1788573948.420888:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573948.420897:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573948.420939:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864127104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573948.420969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573948.420974:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573948.420976:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573948.420981:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5699 00000100:00080000:3.0:1788573948.420985:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573948.420991:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573948.421001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573948.421019:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864127104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573948.421019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864127232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573948.421039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573948.421045:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573948.421047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573948.421052:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5699 00000100:00080000:3.0:1788573948.421057:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573948.421062:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573948.421068:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573948.421081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864127360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573948.421087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573948.421099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573948.421103:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573948.421106:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573948.421111:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5699 00000100:00080000:3.0:1788573948.421116:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573948.421120:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573948.421126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573948.421140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864127488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573948.421145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff92fc67339180 x1875449864127104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573948.421154:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573948.421163:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573948.421169:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573948.421178:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5699 00000100:00080000:3.0:1788573948.421183:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573948.421190:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864127104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573948.421190:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788573948.421207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573948.421210:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573948.421212:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573948.421218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864127104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573948.421229:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864127616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573948.421242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573948.421280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573948.421294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff92fc67339f80 x1875449864127232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573948.421302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864127232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573948.421309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573948.421311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573948.421314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573948.421318:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573948.421331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573948.421338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff92fc6733ad80 x1875449864127360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573948.421345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864127360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573948.421350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573948.421352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573948.421354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573948.421358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573948.421369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573948.421374:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff92fc67339c00 x1875449864127488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573948.421381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864127488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573948.421386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573948.421388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573948.421391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573948.421401:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573948.421412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573948.421418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573948/real 1788573948] req@ffff92fc67339880 x1875449864127616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573948.421426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864127616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573948.421431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573948.421433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573948.421435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573948.868843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573948.932793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573949.892880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573949.956815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573950.916903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573950.980840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573951.940871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573952.004787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573952.964838:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573953.028838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573953.540865:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573953.540874:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573953.540879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573953.540890:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5704 00000100:00080000:2.0:1788573953.540900:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573953.540908:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573953.540958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864127744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573953.540986:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573953.540991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573953.540992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573953.540996:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5704 00000100:00080000:2.0:1788573953.541000:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573953.541004:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573953.541014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573953.541028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864127872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573953.541039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573953.541042:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573953.541044:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573953.541050:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864127744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573953.541052:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5704 00000100:00080000:2.0:1788573953.541060:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573953.541064:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573953.541069:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573953.541082:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864128000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573953.541092:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573953.541095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573953.541097:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573953.541101:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5704 00000100:00080000:2.0:1788573953.541105:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573953.541110:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573953.541115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573953.541120:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573953.541129:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864128128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573953.541140:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573953.541143:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573953.541146:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573953.541151:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5704 00000100:00080000:2.0:1788573953.541155:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573953.541158:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573953.541164:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573953.541169:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff92fc4617fb80 x1875449864127744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573953.541178:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864128256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573953.541208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864127744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573953.541218:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573953.541221:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573953.541225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864127744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573953.541244:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573953.541264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573953.541277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff92fc4617c000 x1875449864127872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573953.541290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864127872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573953.541300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573953.541304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573953.541307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864127872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573953.541313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573953.541346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573953.541355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff92fc4617d880 x1875449864128000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573953.541365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864128000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573953.541380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573953.541384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573953.541388:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573953.541395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573953.541418:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573953.541428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff92fc4617e300 x1875449864128128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573953.541436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864128128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573953.541442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573953.541444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573953.541446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573953.541450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573953.541462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573953.541468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573953/real 1788573953] req@ffff92fc4617ea00 x1875449864128256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573953.541476:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864128256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573953.541480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573953.541482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573953.541485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573953.988888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573954.052787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573955.012842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573955.076846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573956.036887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573956.100829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573957.060864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573957.124823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573958.084866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573958.148824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573958.660827:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.660839:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573958.660844:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.660856:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5709 00000100:00080000:3.0:1788573958.660868:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.660877:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573958.660916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864128384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.660950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.660955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573958.660957:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.660964:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5709 00000100:00080000:3.0:1788573958.660969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.660975:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573958.660983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864128384:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573958.660995:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573958.661015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864128512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.661028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.661032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573958.661035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.661040:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5709 00000100:00080000:3.0:1788573958.661046:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573958.661048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573958.661054:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573958.661085:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff92fc67339500 x1875449864128384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.661099:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864128384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.661107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.661110:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573958.661113:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864128384:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573958.661132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573958.661133:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573958.661154:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864128640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573958.661157:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573958.661170:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff92fc67338380 x1875449864128512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573958.661174:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.661183:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573958.661187:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573958.661189:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864128512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573958.661196:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5709 00000100:00080000:3.0:1788573958.661204:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573958.661209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.661212:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573958.661214:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573958.661221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573958.661233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573958.661239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff92fc67338a80 x1875449864128640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.661247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864128640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.661253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.661255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573958.661257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128640:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788573958.661337:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573958.661345:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573958.661362:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864128768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.661459:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.661463:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573958.661466:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573958.661471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128768:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788573958.661474:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5709 00000100:00080000:3.0:1788573958.661482:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.661488:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788573958.661495:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788573958.661495:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573958.661508:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff92fc6733b800 x1875449864128768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573958.661515:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864128896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573958.661521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864128768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.661543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.661550:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573958.661557:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573958.661572:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573958.661585:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573958.661592:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff92fc6733b100 x1875449864128896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.661599:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864128896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.661605:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.661606:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573958.661609:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864128896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573959.108913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573959.172899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573960.132945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573960.196870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573961.156828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573961.220821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573962.180878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573962.244815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573963.204874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573963.268833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573963.780801:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573963.780809:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573963.780812:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573963.780820:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5714 00000100:00080000:3.0:1788573963.780828:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573963.780834:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573963.780873:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864129024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573963.780902:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573963.780907:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573963.780909:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573963.780914:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5714 00000100:00080000:3.0:1788573963.780920:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573963.780924:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573963.780930:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864129024:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788573963.780942:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573963.780963:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864129152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573963.780976:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573963.780980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573963.780983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573963.780990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573963.780990:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5714 00000100:00080000:3.0:1788573963.780997:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573963.781002:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573963.781015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573963.781029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864129280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573963.781037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573963.781039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573963.781040:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573963.781043:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5714 00000100:00080000:3.0:1788573963.781046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573963.781049:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff92fc67338700 x1875449864129024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573963.781052:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573963.781060:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573963.781084:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864129024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573963.781084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864129408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573963.781113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573963.781115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573963.781117:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788573963.781121:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788573963.781124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864129024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573963.781125:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573963.781133:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5714 00000100:00080000:3.0:1788573963.781136:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573963.781139:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573963.781144:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573963.781148:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573963.781155:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864129536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573963.781169:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573963.781181:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff92fc6733bb80 x1875449864129152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.781191:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864129152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.781198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.781200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573963.781202:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573963.781206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573963.781217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573963.781223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff92fc6733b480 x1875449864129280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.781231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864129280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.781236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.781238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573963.781239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573963.781243:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573963.781253:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573963.781259:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff92fc6733aa00 x1875449864129408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.781266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864129408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.781272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.781274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573963.781276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573963.781285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573963.781295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573963.781301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff92fc67338000 x1875449864129536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.781308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864129536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.781313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.781315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573963.781317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573964.228869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573964.292816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573965.252884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573965.316837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573966.276847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573966.340833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573967.300925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573967.364864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573968.324825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573968.388838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573968.900865:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.900877:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573968.900882:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.900899:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5719 00000100:00080000:3.0:1788573968.900911:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.900921:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573968.900972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864129664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573968.901004:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.901008:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573968.901010:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.901015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5719 00000100:00080000:3.0:1788573968.901020:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.901025:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573968.901038:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573968.901052:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864129792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573968.901062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573968.901064:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864129664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573968.901069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573968.901073:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.901077:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5719 00000100:00080000:3.0:1788573968.901080:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.901083:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573968.901086:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573968.901096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864129920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573968.901109:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.901112:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573968.901113:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.901116:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5719 00000100:00080000:3.0:1788573968.901135:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.901137:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573968.901141:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573968.901153:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864130048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573968.901154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573968.901163:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.901166:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573968.901168:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.901171:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5719 00000100:00080000:3.0:1788573968.901174:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.901177:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573968.901180:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573968.901188:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864130176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573968.901212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff92fc6733a680 x1875449864129664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.901230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864129664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.901240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.901244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573968.901248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864129664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788573968.901285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573968.901303:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573968.901312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff92fc67338e00 x1875449864129792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.901321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864129792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.901328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.901330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573968.901332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573968.901336:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788573968.901347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573968.901354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff92fc6733a300 x1875449864129920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.901361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864129920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.901367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.901368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573968.901370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864129920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788573968.901374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573968.901385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573968.901391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff92fc67338000 x1875449864130048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.901399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864130048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.901404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.901406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573968.901409:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788573968.901412:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788573968.901422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573968.901427:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff92fc6733aa00 x1875449864130176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.901441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864130176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.901446:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.901449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788573968.901451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573969.348958:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573969.412869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573970.372943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573970.436887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573971.396929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573971.460824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573972.420895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573972.484861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573973.444946:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573973.508804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788573974.020893:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573974.020909:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573974.020914:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573974.020925:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5724 00000100:00080000:2.0:1788573974.020937:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573974.020945:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788573974.020992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864130304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573974.021025:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573974.021029:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573974.021030:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573974.021033:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5724 00000100:00080000:2.0:1788573974.021036:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573974.021038:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573974.021048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573974.021058:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864130432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573974.021067:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573974.021074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573974.021075:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573974.021080:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5724 00000100:00100000:0.0:1788573974.021082:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864130304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788573974.021083:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573974.021089:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573974.021094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573974.021105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864130560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573974.021112:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573974.021115:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573974.021116:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573974.021119:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5724 00000100:00080000:2.0:1788573974.021122:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573974.021124:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573974.021129:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573974.021137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864130688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573974.021143:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573974.021146:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573974.021147:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573974.021149:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5724 00000100:00080000:2.0:1788573974.021151:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573974.021153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788573974.021157:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573974.021160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788573974.021167:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864130816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573974.021217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573974/real 1788573974] req@ffff92fc4617d500 x1875449864130304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573974.021233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864130304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573974.021244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573974.021248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573974.021253:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864130304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573974.021259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573974.021276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573974.021286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573974/real 1788573974] req@ffff92fc4617ce00 x1875449864130432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573974.021294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864130432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573974.021304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573974.021307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573974.021309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573974.021332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573974.021343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573974.021349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573974/real 1788573974] req@ffff92fc4617ea00 x1875449864130560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573974.021357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864130560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573974.021363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573974.021365:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573974.021367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573974.021370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573974.021381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573974.021387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573974/real 1788573974] req@ffff92fc4617e300 x1875449864130688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573974.021394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864130688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573974.021399:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573974.021400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573974.021403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573974.021407:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573974.021417:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573974.021422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573974/real 1788573974] req@ffff92fc4617d880 x1875449864130816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573974.021429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864130816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573974.021435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573974.021436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573974.021438:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864130816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573974.468895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573974.532862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573975.492945:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573975.556869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573976.516888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573976.580871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573977.540848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573977.604851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573978.564844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573978.628827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573979.140861:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573979.140870:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573979.140873:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573979.140882:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5729 00000100:00080000:3.0:1788573979.140893:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573979.140903:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573979.140941:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864130944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573979.140971:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573979.140975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573979.140978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573979.140982:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5729 00000100:00080000:3.0:1788573979.140987:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573979.140992:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573979.141002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573979.141016:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864130944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573979.141020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864131072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573979.141034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573979.141037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573979.141039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573979.141043:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5729 00000100:00080000:3.0:1788573979.141048:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573979.141053:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573979.141059:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573979.141071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864131200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573979.141078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573979.141085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573979.141089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573979.141091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573979.141095:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5729 00000100:00080000:3.0:1788573979.141099:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573979.141102:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573979.141109:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573979.141119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573979/real 1788573979] req@ffff92fc6733b480 x1875449864130944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573979.141124:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864131328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573979.141152:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573979.141159:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573979.141162:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573979.141167:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5729 00000100:00080000:3.0:1788573979.141171:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573979.141174:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573979.141179:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864130944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573979.141180:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573979.141194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573979.141199:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788573979.141200:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864131456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573979.141204:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864130944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573979.141231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573979.141251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573979.141260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573979/real 1788573979] req@ffff92fc6733bb80 x1875449864131072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573979.141268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864131072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573979.141274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573979.141276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573979.141278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573979.141282:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573979.141292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573979.141298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573979/real 1788573979] req@ffff92fc67338700 x1875449864131200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573979.141306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864131200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573979.141314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573979.141316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573979.141318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573979.141322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573979.141352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573979.141364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573979/real 1788573979] req@ffff92fc6733b100 x1875449864131328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573979.141377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864131328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573979.141384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573979.141386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573979.141388:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573979.141392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573979.141412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573979.141420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573979/real 1788573979] req@ffff92fc6733b800 x1875449864131456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573979.141430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864131456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573979.141435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573979.141437:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573979.141440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573979.588863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573979.652850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573980.612824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573980.676822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573981.636869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573981.700850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573982.660835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573982.724831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573983.684885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573983.748827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573984.260838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573984.260854:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573984.260860:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573984.260874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5734 00000100:00080000:3.0:1788573984.260888:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573984.260908:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573984.260964:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864131584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573984.261002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573984.261006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573984.261009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573984.261013:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5734 00000100:00080000:3.0:1788573984.261017:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573984.261022:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573984.261035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573984.261065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864131712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573984.261076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573984.261080:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573984.261082:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573984.261086:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5734 00000100:00080000:3.0:1788573984.261091:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573984.261095:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573984.261101:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573984.261116:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864131840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573984.261127:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573984.261131:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573984.261134:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573984.261138:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5734 00000100:00080000:3.0:1788573984.261145:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573984.261147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864131584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573984.261153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573984.261162:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573984.261177:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864131968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573984.261189:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573984.261192:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573984.261194:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573984.261198:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5734 00000100:00080000:3.0:1788573984.261202:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573984.261219:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573984.261225:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573984.261240:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864132096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573984.261244:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573984.261300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff92fc67338a80 x1875449864131584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573984.261325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864131584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573984.261339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573984.261345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573984.261349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864131584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573984.261357:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573984.261378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573984.261388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff92fc67338380 x1875449864131712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573984.261397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864131712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573984.261404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573984.261407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573984.261410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573984.261414:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573984.261431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573984.261441:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff92fc67339500 x1875449864131840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573984.261455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864131840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573984.261464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573984.261468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573984.261472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573984.261500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573984.261530:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573984.261542:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff92fc67339880 x1875449864131968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573984.261556:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864131968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573984.261565:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573984.261568:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573984.261571:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864131968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573984.261576:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573984.261613:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573984.261622:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573984/real 1788573984] req@ffff92fc67339c00 x1875449864132096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573984.261630:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864132096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573984.261635:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573984.261638:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573984.261640:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573984.708923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573984.772878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573985.732914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573985.796884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573986.756940:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573986.820867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573987.780885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573987.844822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573988.804878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573988.868836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573989.380831:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573989.380840:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573989.380843:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573989.380852:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5739 00000100:00080000:3.0:1788573989.380861:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573989.380867:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573989.380900:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864132224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573989.380923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573989.380928:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573989.380930:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573989.380932:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5739 00000100:00080000:3.0:1788573989.380935:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573989.380937:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573989.380944:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573989.380952:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864132352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573989.380959:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573989.380961:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573989.380963:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573989.380965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5739 00000100:00080000:3.0:1788573989.380967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573989.380969:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573989.380972:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573989.380982:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864132480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573989.380983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864132224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788573989.380998:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573989.381000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573989.381001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573989.381003:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5739 00000100:00080000:3.0:1788573989.381013:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573989.381016:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573989.381021:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573989.381030:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864132608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573989.381038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573989.381040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573989.381041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573989.381043:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5739 00000100:00080000:3.0:1788573989.381046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573989.381049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573989.381052:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573989.381056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573989.381064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864132736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573989.381105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff92fc6733ad80 x1875449864132224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573989.381136:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864132224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573989.381145:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573989.381148:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573989.381151:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864132224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573989.381157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573989.381176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573989.381185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff92fc67339f80 x1875449864132352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573989.381193:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864132352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573989.381199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573989.381201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573989.381203:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573989.381223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573989.381233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573989.381240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff92fc67339180 x1875449864132480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573989.381248:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864132480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573989.381257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573989.381259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573989.381261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573989.381265:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573989.381275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573989.381280:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff92fc67339c00 x1875449864132608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573989.381287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864132608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573989.381293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573989.381294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573989.381296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573989.381300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573989.381309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573989.381315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573989/real 1788573989] req@ffff92fc67339880 x1875449864132736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573989.381321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864132736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573989.381334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573989.381336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573989.381339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573989.828832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573989.892844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573990.852864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573990.916801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573991.876891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573991.940831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573992.900845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573992.964776:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573993.924836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573993.988820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573994.500921:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573994.500930:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573994.500934:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573994.500946:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5745 00000100:00080000:3.0:1788573994.500958:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573994.500967:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573994.501002:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864132864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573994.501042:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573994.501047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573994.501050:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573994.501055:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5745 00000100:00080000:3.0:1788573994.501061:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573994.501067:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573994.501079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573994.501092:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864132864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573994.501098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864132992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573994.501113:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573994.501117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573994.501119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573994.501123:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5745 00000100:00080000:3.0:1788573994.501127:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573994.501132:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573994.501137:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573994.501188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788573994.501241:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864133120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573994.501262:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573994.501266:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573994.501269:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573994.501274:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5745 00000100:00080000:3.0:1788573994.501280:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573994.501286:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788573994.501288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133120:192.168.202.132@tcp:38:kworker.0 00000008:00100000:3.0:1788573994.501296:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573994.501316:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864133248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573994.501337:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573994.501342:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573994.501344:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573994.501349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133248:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788573994.501351:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5745 00000100:00080000:3.0:1788573994.501359:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573994.501364:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573994.501370:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573994.501385:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864133376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573994.501427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573994.948864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573995.012830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573995.972918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573996.036882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573996.996870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573997.060845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788573997.573044:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788573997.573098:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788573997.573120:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573997.573169:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573997.573186:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573997.573199:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788573997.573212:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788573997.573229:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff92fc6733b800 x1875449864133248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573997.573263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864133248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573997.573279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573997.573284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573997.573289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573997.573298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff92fc67338a80 x1875449864133120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573997.573319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864133120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573997.573328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573997.573333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573997.573337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573997.573345:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff92fc67338380 x1875449864132992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573997.573357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864132992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573997.573366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573997.573370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573997.573373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864132992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573997.573381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff92fc67339500 x1875449864132864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573997.573394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864132864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573997.573403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573997.573407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573997.573410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864132864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573997.573446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573994/real 1788573997] req@ffff92fc6733b100 x1875449864133376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573997.573459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864133376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573997.573467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573997.573471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573997.573474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788573998.020832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573998.084826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788573999.044863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573999.108840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788573999.620838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.620852:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573999.620858:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.620870:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5750 00000100:00080000:3.0:1788573999.620884:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.620894:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788573999.620942:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864133504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.620980:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.620990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573999.620992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.620997:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5750 00000100:00080000:3.0:1788573999.621001:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.621007:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573999.621022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573999.621033:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864133504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788573999.621054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864133632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.621069:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.621073:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573999.621076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.621081:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5750 00000100:00080000:3.0:1788573999.621086:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.621091:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573999.621097:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573999.621112:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864133760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.621124:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.621128:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573999.621130:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573999.621136:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788573999.621141:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5750 00000100:00080000:3.0:1788573999.621147:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.621152:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573999.621160:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573999.621174:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864133888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.621185:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.621189:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573999.621192:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.621197:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5750 00000100:00080000:3.0:1788573999.621202:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573999.621207:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff92fc67338700 x1875449864133504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573999.621213:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788573999.621224:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573999.621241:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864134016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573999.621244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864133504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573999.621274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573999.621280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573999.621285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864133504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788573999.621322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573999.621351:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573999.621365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff92fc6733bb80 x1875449864133632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573999.621380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864133632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573999.621390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573999.621393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573999.621397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573999.621404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788573999.621429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573999.621443:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff92fc6733b480 x1875449864133760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573999.621456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864133760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573999.621465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573999.621469:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573999.621472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788573999.621478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573999.621500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573999.621512:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff92fc6733aa00 x1875449864133888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573999.621525:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864133888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573999.621533:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573999.621537:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573999.621540:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864133888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788573999.621546:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788573999.621568:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788573999.621580:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff92fc67338000 x1875449864134016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573999.621593:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864134016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573999.621602:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573999.621605:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788573999.621609:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574000.068899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574000.132879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574001.092870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574001.156828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574002.116893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574002.180800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574003.140898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574003.204833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574004.164885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574004.228816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574004.740838:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574004.740848:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574004.740853:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574004.740864:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5755 00000100:00080000:3.0:1788574004.740875:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574004.740882:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574004.740921:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864134144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574004.740950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574004.740955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574004.740957:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574004.740962:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5755 00000100:00080000:3.0:1788574004.740966:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574004.740971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574004.740981:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574004.740999:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864134272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574004.741000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864134144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574004.741028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574004.741032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574004.741034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574004.741038:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5755 00000100:00080000:3.0:1788574004.741043:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574004.741047:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574004.741053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574004.741066:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864134400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574004.741076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574004.741079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574004.741084:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574004.741087:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574004.741092:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5755 00000100:00080000:3.0:1788574004.741097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574004.741102:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574004.741108:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574004.741121:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864134528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574004.741128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff92fc6733a300 x1875449864134144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574004.741137:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574004.741142:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574004.741144:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574004.741148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864134144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574004.741152:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5755 00000100:00080000:3.0:1788574004.741160:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574004.741164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574004.741169:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788574004.741172:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574004.741177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864134144:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788574004.741178:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574004.741194:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864134656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574004.741199:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574004.741224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574004.741236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff92fc67338e00 x1875449864134272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.741245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864134272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.741251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.741253:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574004.741256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574004.741260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574004.741271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574004.741279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff92fc6733a680 x1875449864134400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.741286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864134400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.741292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.741293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574004.741295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574004.741299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574004.741310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574004.741316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff92fc67338000 x1875449864134528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.741324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864134528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.741328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.741330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574004.741332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574004.741341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574004.741352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574004.741358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff92fc6733aa00 x1875449864134656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.741365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864134656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.741371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.741373:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574004.741376:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574005.188907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574005.252788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574006.212863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574006.276777:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574007.236825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574007.300815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574008.260811:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574008.324799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574009.284822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574009.348799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574009.860892:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.860903:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574009.860909:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.860921:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5760 00000100:00080000:2.0:1788574009.860933:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.860941:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574009.860989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864134784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.861021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.861026:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574009.861028:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.861031:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5760 00000100:00080000:2.0:1788574009.861034:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.861038:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574009.861052:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574009.861069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864134912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.861085:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574009.861087:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864134784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574009.861092:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574009.861099:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.861105:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5760 00000100:00080000:2.0:1788574009.861109:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.861114:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574009.861118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574009.861132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864135040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.861148:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.861152:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574009.861154:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.861158:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5760 00000100:00080000:2.0:1788574009.861161:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574009.861168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574009.861168:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574009.861184:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574009.861200:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864135168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.861212:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.861216:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574009.861218:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.861225:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5760 00000100:00100000:0.0:1788574009.861226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff92fc4617c000 x1875449864134784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574009.861238:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.861245:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574009.861251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864134784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574009.861251:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574009.861273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788574009.861276:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864135296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574009.861285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574009.861300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864134784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574009.861327:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574009.861346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574009.861356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff92fc4617fb80 x1875449864134912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574009.861364:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864134912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574009.861370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574009.861372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574009.861374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864134912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574009.861378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574009.861388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574009.861394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff92fc4617c380 x1875449864135040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574009.861401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864135040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574009.861406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574009.861408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574009.861410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574009.861413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574009.861424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574009.861430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff92fc4617d180 x1875449864135168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574009.861437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864135168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574009.861442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574009.861444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574009.861447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574009.861450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574009.861461:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574009.861467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574009] req@ffff92fc4617e680 x1875449864135296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574009.861473:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864135296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574009.861478:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574009.861480:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574009.861482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574010.308897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574010.372815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574011.332905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574011.396827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574012.356887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574012.420775:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574013.380839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574013.444806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574014.404923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574014.468816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574014.980863:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574014.980878:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574014.980885:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574014.980899:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5765 00000100:00080000:1.0:1788574014.980914:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574014.980923:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574014.980966:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864135424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574014.980996:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574014.981000:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574014.981003:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574014.981007:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5765 00000100:00080000:1.0:1788574014.981011:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574014.981015:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574014.981028:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574014.981044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864135552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574014.981055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574014.981059:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574014.981061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574014.981062:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864135424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574014.981066:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5765 00000100:00080000:1.0:1788574014.981071:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574014.981077:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574014.981083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574014.981093:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864135680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574014.981105:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574014.981107:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574014.981109:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574014.981112:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5765 00000100:00080000:1.0:1788574014.981114:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574014.981116:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574014.981120:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574014.981129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864135808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574014.981135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574014.981138:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574014.981139:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574014.981141:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5765 00000100:00080000:1.0:1788574014.981156:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574014.981159:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788574014.981164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788574014.981164:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574014.981177:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864135936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574014.981221:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff92fc6728bb80 x1875449864135424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.981244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864135424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.981257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.981262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574014.981267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864135424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574014.981295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574014.981316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574014.981328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff92fc67289500 x1875449864135552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.981342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864135552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.981351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.981355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574014.981359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574014.981365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574014.981381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574014.981390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff92fc67289180 x1875449864135680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.981409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864135680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.981418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.981421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574014.981424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574014.981430:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574014.981447:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574014.981456:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff92fc67288700 x1875449864135808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.981468:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864135808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.981477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.981479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574014.981483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574014.981488:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574014.981503:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574014.981511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff92fc67288a80 x1875449864135936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.981523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864135936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.981538:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.981541:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574014.981545:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864135936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574015.428900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574015.492832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574016.452853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574016.516819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574017.476842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574017.540822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574018.500852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574018.564861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574019.524851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574019.588860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574020.100865:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574020.100877:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574020.100882:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574020.100892:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5770 00000100:00080000:1.0:1788574020.100905:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574020.100912:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574020.100979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864136064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574020.101010:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574020.101013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574020.101015:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574020.101018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5770 00000100:00080000:1.0:1788574020.101021:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574020.101024:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574020.101036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574020.101048:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864136192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574020.101057:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574020.101060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574020.101063:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574020.101064:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864136064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574020.101068:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5770 00000100:00080000:1.0:1788574020.101073:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574020.101077:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574020.101080:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574020.101089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864136320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574020.101096:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574020.101099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574020.101101:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574020.101104:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5770 00000100:00080000:1.0:1788574020.101106:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574020.101109:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574020.101112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574020.101119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864136448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574020.101126:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574020.101128:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574020.101129:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574020.101132:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5770 00000100:00080000:1.0:1788574020.101142:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574020.101144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574020.101148:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574020.101152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574020.101161:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864136576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574020.101203:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574020/real 1788574020] req@ffff92fc6728b100 x1875449864136064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574020.101222:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864136064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574020.101231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574020.101234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574020.101237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864136064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574020.101261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574020.101277:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574020.101287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574020/real 1788574020] req@ffff92fc67288380 x1875449864136192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574020.101299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864136192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574020.101308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574020.101312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574020.101316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574020.101322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574020.101345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574020.101355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574020/real 1788574020] req@ffff92fc67289c00 x1875449864136320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574020.101364:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864136320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574020.101369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574020.101372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574020.101374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574020.101378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574020.101392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574020.101398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574020/real 1788574020] req@ffff92fc6728a680 x1875449864136448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574020.101405:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864136448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574020.101411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574020.101414:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574020.101416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574020.101419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574020.101429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574020.101435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574020/real 1788574020] req@ffff92fc6728b800 x1875449864136576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574020.101442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864136576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574020.101448:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574020.101449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574020.101452:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574020.548900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574020.612871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574021.572857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574021.636850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574022.596878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574022.660849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574023.620878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574023.684801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574024.644868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574024.708848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574025.220882:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574025.220891:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574025.220895:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574025.220905:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5775 00000100:00080000:0.0:1788574025.220914:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574025.220921:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574025.220959:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864136704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574025.220987:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574025.220991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574025.220993:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574025.220997:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5775 00000100:00080000:0.0:1788574025.221000:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574025.221004:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574025.221022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574025.221031:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864136704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574025.221036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864136832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574025.221045:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574025.221047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574025.221049:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574025.221052:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5775 00000100:00080000:0.0:1788574025.221055:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574025.221057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574025.221060:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574025.221068:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864136960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574025.221079:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574025.221082:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574025.221083:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574025.221085:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5775 00000100:00080000:0.0:1788574025.221088:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574025.221090:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788574025.221095:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574025.221096:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574025.221105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864137088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574025.221112:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574025.221114:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574025.221116:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574025.221118:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5775 00000100:00080000:0.0:1788574025.221120:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574025.221123:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574025.221125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574025.221134:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff92fc6704b100 x1875449864136704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574025.221136:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864137216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574025.221158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864136704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574025.221167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574025.221170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574025.221173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864136704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574025.221191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574025.221206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574025.221215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff92fc67049500 x1875449864136832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574025.221224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864136832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574025.221230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574025.221233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574025.221235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574025.221238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574025.221248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574025.221254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff92fc67049180 x1875449864136960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574025.221261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864136960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574025.221267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574025.221269:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574025.221271:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864136960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574025.221274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574025.221284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574025.221290:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff92fc67049c00 x1875449864137088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574025.221302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864137088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574025.221307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574025.221309:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574025.221310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574025.221314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574025.221330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574025.221340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574025/real 1788574025] req@ffff92fc67048e00 x1875449864137216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574025.221352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864137216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574025.221360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574025.221363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574025.221367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574025.668884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574025.732835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574026.692864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574026.756832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574027.716881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574027.780794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574028.740918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574028.804916:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574029.764901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574029.828788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574030.340858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.340864:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574030.340867:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.340876:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5780 00000100:00080000:3.0:1788574030.340892:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574030.340899:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574030.340929:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864137344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574030.340957:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.340960:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574030.340962:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.340966:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5780 00000100:00080000:3.0:1788574030.340969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574030.340971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574030.340980:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574030.340992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864137472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574030.341001:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864137344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574030.341005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.341011:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574030.341012:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.341016:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5780 00000100:00080000:3.0:1788574030.341019:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574030.341024:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574030.341029:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574030.341042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864137600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574030.341052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.341056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574030.341058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574030.341064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574030.341068:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5780 00000100:00080000:3.0:1788574030.341075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574030.341109:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff92fc6733b480 x1875449864137344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.341128:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864137344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574030.341136:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574030.341146:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574030.341150:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.341157:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574030.341160:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864137728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574030.341167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864137344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574030.341172:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574030.341177:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574030.341179:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574030.341183:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5780 00000100:00080000:3.0:1788574030.341187:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574030.341189:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574030.341193:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574030.341199:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574030.341202:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864137856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574030.341229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574030.341253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff92fc6733bb80 x1875449864137472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.341268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864137472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.341277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.341281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574030.341284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574030.341291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574030.341313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574030.341323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff92fc67338700 x1875449864137600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.341330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864137600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.341336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.341339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574030.341341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574030.341345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574030.341357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574030.341365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff92fc6733b100 x1875449864137728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.341372:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864137728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.341378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.341380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574030.341382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574030.341394:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574030.341416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574030.341429:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574030/real 1788574030] req@ffff92fc67339500 x1875449864137856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574030.341443:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864137856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574030.341451:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574030.341456:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574030.341460:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864137856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574030.788853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574030.852842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574031.812876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574031.876822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574032.836833:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574032.900875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574033.860891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574033.924862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574034.884913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574034.948920:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574035.460861:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574035.460869:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574035.460873:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574035.460882:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5786 00000100:00080000:3.0:1788574035.460892:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574035.460899:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574035.460933:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864137984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574035.460958:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574035.460961:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574035.460963:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574035.460965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5786 00000100:00080000:3.0:1788574035.460967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574035.460969:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574035.460977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574035.460987:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864138112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574035.460994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574035.460997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574035.460998:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574035.461000:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5786 00000100:00080000:3.0:1788574035.461004:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574035.461008:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574035.461010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864137984:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788574035.461012:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574035.461022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864138240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574035.461029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574035.461031:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574035.461033:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574035.461035:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5786 00000100:00080000:3.0:1788574035.461037:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574035.461040:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574035.461043:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574035.461050:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864138368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574035.461056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574035.461059:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574035.461060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574035.461062:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5786 00000100:00080000:3.0:1788574035.461071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574035.461074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574035.461077:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574035.461082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574035.461090:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864138496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574035.461116:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff92fc67338380 x1875449864137984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.461132:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864137984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.461141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.461144:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574035.461146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864137984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574035.461152:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574035.461167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574035.461186:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff92fc67338a80 x1875449864138112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.461195:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864138112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.461201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.461203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574035.461205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574035.461224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574035.461236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574035.461243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff92fc6733b800 x1875449864138240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.461250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864138240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.461260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.461262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574035.461271:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574035.461275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574035.461287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574035.461293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff92fc67339880 x1875449864138368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.461301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864138368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.461306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.461309:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574035.461311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574035.461314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574035.461323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574035.461329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574035/real 1788574035] req@ffff92fc67339c00 x1875449864138496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574035.461335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864138496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574035.461340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574035.461342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574035.461345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574035.908945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574035.972920:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574036.932833:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574036.996841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574037.956906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574038.020884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574038.980862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574039.044830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574040.004902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574040.068840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574040.580828:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574040.580838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574040.580843:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574040.580853:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5791 00000100:00080000:3.0:1788574040.580863:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574040.580878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574040.580914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864138624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574040.580941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574040.580945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574040.580947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574040.580952:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5791 00000100:00080000:3.0:1788574040.580957:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574040.580961:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574040.580972:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574040.580979:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864138624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574040.580989:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864138752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574040.581003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574040.581012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574040.581015:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574040.581019:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5791 00000100:00080000:3.0:1788574040.581024:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574040.581028:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574040.581034:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574040.581047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864138880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574040.581053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574040.581065:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574040.581068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574040.581070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574040.581076:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5791 00000100:00080000:3.0:1788574040.581080:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574040.581084:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574040.581090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574040.581102:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff92fc67339180 x1875449864138624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574040.581107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864139008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574040.581125:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574040.581129:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788574040.581131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864138624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574040.581132:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574040.581142:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5791 00000100:00080000:3.0:1788574040.581150:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574040.581153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574040.581157:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:1.0:1788574040.581162:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:3.0:1788574040.581165:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574040.581167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864138624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574040.581184:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864139136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574040.581193:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574040.581219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574040.581230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff92fc67339f80 x1875449864138752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.581239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864138752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.581245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.581248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574040.581250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574040.581256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574040.581274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574040.581287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff92fc6733ad80 x1875449864138880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.581300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864138880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.581309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.581312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574040.581316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864138880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574040.581323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574040.581345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574040.581358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff92fc67339c00 x1875449864139008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.581371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864139008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.581379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.581383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574040.581386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574040.581404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574040.581423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574040.581432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff92fc67339880 x1875449864139136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574040.581440:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864139136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574040.581445:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574040.581447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574040.581450:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574041.028874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574041.092832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574042.052892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574042.116832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574043.076893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574043.140846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574044.101018:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574044.164884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574045.124922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574045.188801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574045.700890:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574045.700899:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574045.700902:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574045.700911:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5796 00000100:00080000:2.0:1788574045.700920:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574045.700926:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574045.700958:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864139264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574045.700982:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574045.700985:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574045.700987:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574045.700990:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5796 00000100:00080000:2.0:1788574045.700992:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574045.700994:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574045.701002:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574045.701010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864139392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574045.701017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574045.701020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574045.701021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574045.701024:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5796 00000100:00080000:2.0:1788574045.701026:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574045.701029:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574045.701032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574045.701046:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864139520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574045.701048:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864139264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574045.701062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574045.701067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574045.701068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574045.701071:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5796 00000100:00080000:2.0:1788574045.701075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574045.701078:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574045.701082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574045.701096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864139648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574045.701103:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574045.701105:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574045.701107:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574045.701109:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5796 00000100:00080000:2.0:1788574045.701115:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574045.701118:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574045.701122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574045.701128:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864139776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574045.701142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574045.701191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff92fc4617dc00 x1875449864139264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.701208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864139264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.701218:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.701222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574045.701225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864139264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574045.701231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574045.701248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574045.701256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff92fc4617ca80 x1875449864139392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.701265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864139392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.701271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.701273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574045.701275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574045.701302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574045.701322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574045.701334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff92fc4617c700 x1875449864139520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.701347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864139520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.701356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.701359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574045.701363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574045.701368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574045.701386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574045.701394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff92fc4617f800 x1875449864139648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.701401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864139648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.701407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.701409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574045.701411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574045.701414:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574045.701425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574045.701431:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff92fc4617f100 x1875449864139776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574045.701440:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864139776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574045.701445:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574045.701446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574045.701448:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864139776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574046.148984:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574046.212893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574047.172898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574047.236798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574048.196956:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574048.260888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574049.220947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574049.284816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574050.244921:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574050.308879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574050.820876:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.820891:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574050.820896:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.820910:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5801 00000100:00080000:2.0:1788574050.820925:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.820935:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574050.820982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864139904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574050.821013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.821019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574050.821020:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.821023:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5801 00000100:00080000:2.0:1788574050.821026:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.821028:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574050.821039:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574050.821048:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864140032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574050.821055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.821058:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574050.821060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.821062:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5801 00000100:00080000:2.0:1788574050.821064:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.821067:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574050.821070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574050.821079:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864140160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574050.821086:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.821088:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574050.821090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574050.821095:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864139904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574050.821095:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5801 00000100:00080000:2.0:1788574050.821098:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.821103:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574050.821108:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574050.821116:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864140288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574050.821122:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574050.821125:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574050.821126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574050.821129:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5801 00000100:00080000:2.0:1788574050.821139:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574050.821142:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574050.821146:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574050.821154:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864140416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574050.821172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574050.821229:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff92fc4617f480 x1875449864139904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.821253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864139904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.821270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.821274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574050.821277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864139904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574050.821283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574050.821306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574050.821320:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff92fc4617ed80 x1875449864140032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.821333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864140032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.821348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.821352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574050.821355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574050.821388:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574050.821409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574050.821417:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff92fc4617f100 x1875449864140160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.821425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864140160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.821431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.821434:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574050.821436:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574050.821439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574050.821452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574050.821458:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff92fc4617f800 x1875449864140288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.821465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864140288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.821470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.821471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574050.821474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574050.821478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574050.821489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574050.821494:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff92fc4617c700 x1875449864140416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574050.821501:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864140416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574050.821506:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574050.821508:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574050.821509:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574051.268898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574051.332826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574052.292826:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574052.356837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574053.316868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574053.380802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574054.340851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574054.404823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574055.364951:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574055.428891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574055.940882:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.940892:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574055.940897:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.940908:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5806 00000100:00080000:3.0:1788574055.940919:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.940927:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574055.940970:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864140544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.940995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.940998:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.941000:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.941003:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5806 00000100:00080000:3.0:1788574055.941005:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.941008:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574055.941017:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574055.941027:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864140672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.941035:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.941038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574055.941039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.941042:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5806 00000100:00080000:3.0:1788574055.941044:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.941046:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574055.941049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574055.941059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864140800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574055.941060:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864140544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574055.941079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.941082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.941083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.941086:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5806 00000100:00080000:3.0:1788574055.941088:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.941090:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574055.941093:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574055.941102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864140928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.941109:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.941111:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574055.941113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.941115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5806 00000100:00080000:3.0:1788574055.941117:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.941119:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574055.941122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574055.941129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864141056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574055.941151:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574055.941209:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff92fc6733b800 x1875449864140544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.941232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864140544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.941246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.941250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574055.941254:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864140544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574055.941263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574055.941287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574055.941302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff92fc67338a80 x1875449864140672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.941315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864140672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.941324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.941327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574055.941331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574055.941360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574055.941380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574055.941393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff92fc67338380 x1875449864140800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.941404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864140800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.941413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.941416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574055.941419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574055.941425:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574055.941443:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574055.941453:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff92fc67339500 x1875449864140928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.941464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864140928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.941472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.941475:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574055.941478:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864140928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574055.941483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574055.941500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574055.941510:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff92fc6733b100 x1875449864141056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574055.941521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864141056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574055.941529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574055.941532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574055.941535:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574056.388879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574056.452892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574057.412873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574057.476840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574058.436815:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574058.500814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574059.460873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574059.524837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574060.484822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574060.548822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574061.060840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574061.060853:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574061.060857:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574061.060868:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5811 00000100:00080000:3.0:1788574061.060880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574061.060888:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574061.060929:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864141184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574061.060960:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574061.060964:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574061.060966:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574061.060970:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5811 00000100:00080000:3.0:1788574061.060974:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574061.060978:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574061.060989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574061.061005:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864141312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574061.061012:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864141184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574061.061017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574061.061021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574061.061022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574061.061026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5811 00000100:00080000:3.0:1788574061.061029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574061.061032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574061.061035:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574061.061045:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864141440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574061.061052:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574061.061055:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574061.061056:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574061.061059:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5811 00000100:00080000:3.0:1788574061.061061:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574061.061063:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574061.061067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574061.061074:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864141568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574061.061079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574061.061082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574061.061083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574061.061086:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5811 00000100:00080000:3.0:1788574061.061088:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574061.061091:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574061.061094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574061.061104:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864141696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574061.061117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574061.061145:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574061.061172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574061.061198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574061.508857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574061.572830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574062.532872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574062.596830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574063.556846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574063.620820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574064.132985:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574064.133031:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574064.133048:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574064.133089:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574064.133097:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574064.133104:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574064.133115:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788574064.133144:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574061/real 1788574064] req@ffff92fc6733bb80 x1875449864141312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574064.133187:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864141312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574064.133200:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574064.133206:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574064.133210:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574064.133219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574061/real 1788574064] req@ffff92fc6733b480 x1875449864141440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574064.133234:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864141440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574064.133243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574064.133247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574064.133251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574064.133258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574061/real 1788574064] req@ffff92fc6733aa00 x1875449864141568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574064.133268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864141568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574064.133273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574064.133276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574064.133279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574064.133286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574061/real 1788574064] req@ffff92fc67338000 x1875449864141696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574064.133298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864141696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574064.133305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574064.133308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574064.133311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141696:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574064.133318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574061/real 1788574064] req@ffff92fc67338700 x1875449864141184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574064.133330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864141184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574064.133338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574064.133341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574064.133344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864141184:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788574064.580849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574064.644815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574065.604857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574065.668828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574066.180849:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.180860:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574066.180865:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.180875:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5816 00000100:00080000:3.0:1788574066.180895:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.180904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574066.180943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864141824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574066.180970:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.180974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574066.180975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.180978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5816 00000100:00080000:3.0:1788574066.180981:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.180983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574066.180993:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574066.181002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864141952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574066.181010:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.181013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574066.181015:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.181018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5816 00000100:00080000:3.0:1788574066.181021:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.181025:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574066.181031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574066.181044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864142080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574066.181058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864141824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574066.181062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.181076:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574066.181080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.181087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5816 00000100:00080000:3.0:1788574066.181093:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.181098:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574066.181105:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574066.181122:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864142208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574066.181133:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574066.181137:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574066.181138:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574066.181142:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5816 00000100:00080000:3.0:1788574066.181148:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574066.181153:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574066.181158:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574066.181165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574066.181180:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864142336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574066.181232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff92fc6733a680 x1875449864141824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.181263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864141824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.181276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.181282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574066.181290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864141824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574066.181303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574066.181336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574066.181351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff92fc67338e00 x1875449864141952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.181363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864141952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.181374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.181379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574066.181382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864141952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574066.181425:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574066.181456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574066.181469:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff92fc6733a300 x1875449864142080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.181481:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864142080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.181490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.181494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574066.181497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574066.181506:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574066.181533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574066.181550:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff92fc67338700 x1875449864142208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.181563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864142208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.181583:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.181587:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574066.181590:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574066.181601:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574066.181631:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574066.181643:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574066/real 1788574066] req@ffff92fc67338000 x1875449864142336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574066.181655:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864142336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574066.181665:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574066.181670:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574066.181674:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574066.628908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574066.692820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574067.652938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574067.716865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574068.676866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574068.740788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574069.700853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574069.764815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574070.724926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574070.788836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574071.300812:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.300823:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574071.300828:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.300838:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5821 00000100:00080000:3.0:1788574071.300848:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.300855:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574071.300887:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864142464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574071.300910:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.300915:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574071.300916:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.300919:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5821 00000100:00080000:3.0:1788574071.300921:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.300924:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574071.300933:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574071.300947:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864142592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574071.300954:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864142464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574071.300965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.300970:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574071.300972:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.300977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5821 00000100:00080000:3.0:1788574071.300980:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.300983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574071.300988:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574071.300998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864142720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574071.301005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.301007:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574071.301008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.301011:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5821 00000100:00080000:3.0:1788574071.301014:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.301016:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574071.301020:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574071.301026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574071.301029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864142848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574071.301037:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574071.301040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574071.301042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574071.301044:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5821 00000100:00080000:3.0:1788574071.301047:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574071.301049:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574071.301052:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574071.301059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864142976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574071.301074:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff92fc6733aa00 x1875449864142464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.301090:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864142464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.301102:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.301105:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574071.301108:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864142464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574071.301126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574071.301147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574071.301156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff92fc6733b480 x1875449864142592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.301164:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864142592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.301170:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.301172:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574071.301176:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574071.301182:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574071.301201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574071.301213:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff92fc6733bb80 x1875449864142720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.301226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864142720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.301234:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.301236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574071.301239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574071.301243:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574071.301259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574071.301266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff92fc6733b100 x1875449864142848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.301274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864142848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.301279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.301281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574071.301283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574071.301287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574071.301296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574071.301302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574071/real 1788574071] req@ffff92fc67339500 x1875449864142976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574071.301308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864142976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574071.301313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574071.301315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574071.301317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864142976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574071.748855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574071.812823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574072.772864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574072.836848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574073.796860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574073.860830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574074.820854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574074.884843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574075.844838:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574075.908791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574076.420844:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574076.420855:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574076.420860:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574076.420870:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5827 00000100:00080000:3.0:1788574076.420882:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574076.420890:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574076.420936:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864143104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574076.420965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574076.420969:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574076.420970:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574076.420973:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5827 00000100:00080000:3.0:1788574076.420976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574076.420978:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574076.420989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574076.421000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864143232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574076.421008:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574076.421010:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574076.421011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574076.421013:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5827 00000100:00080000:3.0:1788574076.421016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574076.421018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574076.421020:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574076.421027:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864143360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574076.421034:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574076.421037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574076.421038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574076.421040:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5827 00000100:00080000:3.0:1788574076.421045:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574076.421052:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864143104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574076.421052:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574076.421064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574076.421073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864143488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574076.421084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574076.421087:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574076.421088:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574076.421093:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5827 00000100:00080000:3.0:1788574076.421098:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574076.421102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574076.421107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574076.421121:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864143616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574076.421131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574076.421174:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff92fc67338380 x1875449864143104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.421189:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864143104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.421198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.421201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574076.421204:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864143104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574076.421210:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574076.421225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574076.421233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff92fc67338a80 x1875449864143232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.421241:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864143232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.421247:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.421249:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574076.421251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574076.421272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574076.421282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574076.421289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff92fc6733b800 x1875449864143360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.421296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864143360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.421305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.421307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574076.421310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574076.421313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574076.421323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574076.421329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff92fc67339880 x1875449864143488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.421336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864143488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.421341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.421343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574076.421346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574076.421349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574076.421359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574076.421365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574076/real 1788574076] req@ffff92fc67339c00 x1875449864143616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574076.421372:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864143616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574076.421377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574076.421378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574076.421381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574076.868869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574076.932795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574077.892828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574077.956789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574078.916856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574078.980794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574079.940815:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574080.004802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574080.964839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574081.028844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574081.540785:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574081.540793:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574081.540797:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574081.540806:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5832 00000100:00080000:3.0:1788574081.540815:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574081.540822:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574081.540855:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864143744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574081.540878:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574081.540881:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574081.540882:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574081.540885:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5832 00000100:00080000:3.0:1788574081.540887:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574081.540889:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574081.540898:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574081.540906:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864143872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574081.540914:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574081.540916:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574081.540918:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574081.540920:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5832 00000100:00080000:3.0:1788574081.540922:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574081.540924:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574081.540929:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574081.540930:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864143744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574081.540937:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864144000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574081.540945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574081.540948:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574081.540949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574081.540951:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5832 00000100:00080000:3.0:1788574081.540953:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574081.540955:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574081.540959:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574081.540968:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864144128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574081.540974:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574081.540976:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574081.540978:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574081.540980:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5832 00000100:00080000:3.0:1788574081.540983:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574081.540985:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574081.540989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574081.540993:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574081.540999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864144256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574081.541035:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff92fc6733ad80 x1875449864143744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574081.541057:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864143744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574081.541069:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574081.541073:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574081.541077:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864143744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574081.541084:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574081.541107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574081.541121:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff92fc67339f80 x1875449864143872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574081.541135:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864143872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574081.541144:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574081.541147:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574081.541149:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864143872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574081.541170:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574081.541185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574081.541194:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff92fc67339180 x1875449864144000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574081.541202:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864144000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574081.541207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574081.541209:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574081.541211:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574081.541214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574081.541225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574081.541230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff92fc67339c00 x1875449864144128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574081.541237:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864144128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574081.541242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574081.541244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574081.541246:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574081.541250:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574081.541259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574081.541265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574081/real 1788574081] req@ffff92fc67339880 x1875449864144256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574081.541272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864144256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574081.541278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574081.541281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574081.541285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574081.988878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574082.052803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574083.012853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574083.076830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574084.036891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574084.100800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574085.060852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574085.124829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574086.084859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574086.148799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574086.660859:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.660868:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574086.660872:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.660880:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5837 00000100:00080000:2.0:1788574086.660889:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.660896:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574086.660929:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864144384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.660951:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.660955:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574086.660956:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.660959:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5837 00000100:00080000:2.0:1788574086.660962:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.660965:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574086.660975:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574086.660990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864144512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.661002:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574086.661004:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864144384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574086.661015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574086.661019:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.661028:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5837 00000100:00080000:2.0:1788574086.661032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.661037:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574086.661043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574086.661056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864144640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.661064:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.661066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574086.661068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.661073:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5837 00000800:00000100:1.0:1788574086.661074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574086.661077:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.661081:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574086.661086:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574086.661094:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864144768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.661101:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.661103:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574086.661104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.661107:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5837 00000100:00080000:2.0:1788574086.661110:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.661112:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574086.661115:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574086.661121:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff92fc4617ca80 x1875449864144384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574086.661124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864144896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574086.661162:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864144384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.661174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.661179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574086.661183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864144384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574086.661209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574086.661230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574086.661242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff92fc4617dc00 x1875449864144512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.661255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864144512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.661263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.661267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574086.661271:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574086.661277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574086.661294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574086.661304:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff92fc4617e680 x1875449864144640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.661316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864144640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.661324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.661327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574086.661330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574086.661336:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574086.661353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574086.661363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff92fc4617d180 x1875449864144768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.661374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864144768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.661382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.661386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574086.661389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574086.661395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574086.661446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574086.661459:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff92fc4617c380 x1875449864144896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.661472:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864144896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.661479:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.661483:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574086.661487:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864144896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574087.108835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574087.172833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574088.132895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574088.196783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574089.156875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574089.220815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574090.180850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574090.244789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574091.204834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574091.268820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574091.780799:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574091.780807:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574091.780811:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574091.780820:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5842 00000100:00080000:2.0:1788574091.780829:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574091.780835:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574091.780873:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864145024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574091.780898:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574091.780902:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574091.780903:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574091.780906:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5842 00000100:00080000:2.0:1788574091.780908:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574091.780910:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574091.780919:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574091.780928:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864145152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574091.780936:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574091.780939:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574091.780941:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574091.780947:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864145024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574091.780947:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5842 00000100:00080000:2.0:1788574091.780950:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574091.780957:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574091.780962:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574091.780969:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864145280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574091.780976:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574091.780978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574091.780980:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574091.780982:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5842 00000100:00080000:2.0:1788574091.780985:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574091.780987:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574091.780991:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574091.780999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864145408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574091.781004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574091.781008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574091.781009:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574091.781013:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5842 00000100:00080000:2.0:1788574091.781017:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574091.781020:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574091.781025:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574091.781032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574091.781047:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864145536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574091.781061:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff92fc4617fb80 x1875449864145024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.781078:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864145024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.781087:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.781090:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574091.781093:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864145024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574091.781112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574091.781129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574091.781137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff92fc4617c000 x1875449864145152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.781145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864145152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.781155:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.781157:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574091.781160:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574091.781163:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574091.781173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574091.781179:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff92fc4617d880 x1875449864145280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.781186:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864145280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.781192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.781193:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574091.781195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574091.781198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574091.781208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574091.781214:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff92fc4617e300 x1875449864145408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.781221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864145408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.781226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.781228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574091.781231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574091.781234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574091.781274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574091.781281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff92fc4617ea00 x1875449864145536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574091.781289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864145536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574091.781294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574091.781296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574091.781338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574092.228866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574092.292799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574093.252811:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574093.316785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574094.276867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574094.340846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574095.300802:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574095.364813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574096.324876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574096.388879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574096.900818:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.900827:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574096.900830:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.900839:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5847 00000100:00080000:3.0:1788574096.900847:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.900854:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574096.900885:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864145664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574096.900906:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.900909:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574096.900911:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.900913:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5847 00000100:00080000:3.0:1788574096.900916:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.900918:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574096.900926:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574096.900935:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864145792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574096.900947:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864145664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574096.900949:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.900955:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574096.900958:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.900962:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5847 00000100:00080000:3.0:1788574096.900964:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.900966:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574096.900969:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574096.900977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864145920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574096.900982:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.900985:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574096.900986:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.900989:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5847 00000100:00080000:3.0:1788574096.900991:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.900993:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574096.900997:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574096.901003:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864146048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574096.901009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574096.901011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574096.901012:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574096.901014:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5847 00000100:00080000:3.0:1788574096.901017:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574096.901022:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574096.901024:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574096.901026:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574096.901035:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864146176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574096.901073:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff92fc6733b800 x1875449864145664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.901089:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864145664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.901097:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.901100:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574096.901103:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864145664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574096.901122:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574096.901139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574096.901148:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff92fc67338a80 x1875449864145792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.901157:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864145792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.901163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.901165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574096.901168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574096.901171:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574096.901182:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574096.901188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff92fc67338380 x1875449864145920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.901196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864145920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.901202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.901204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574096.901206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864145920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574096.901209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574096.901220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574096.901226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff92fc67339500 x1875449864146048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.901233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864146048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.901239:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.901241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574096.901244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574096.901247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574096.901257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574096.901262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff92fc6733b100 x1875449864146176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.901273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864146176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.901281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.901284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574096.901288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574097.348829:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574097.412815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574098.372836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574098.436832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574099.396850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574099.460863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574100.420904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574100.484835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574101.444817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574101.508805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574102.020873:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574102.020881:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574102.020885:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574102.020895:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5852 00000100:00080000:2.0:1788574102.020905:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574102.020912:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574102.020942:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864146304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574102.020967:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574102.020972:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574102.020974:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574102.020978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5852 00000100:00080000:2.0:1788574102.020982:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574102.020985:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574102.020993:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574102.021002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864146304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574102.021007:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864146432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574102.021018:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574102.021022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574102.021025:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574102.021029:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5852 00000100:00080000:2.0:1788574102.021034:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574102.021039:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574102.021045:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574102.021059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864146560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574102.021069:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574102.021074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574102.021076:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574102.021080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574102.021086:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5852 00000100:00080000:2.0:1788574102.021091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574102.021094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574102.021100:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574102.021109:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864146688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574102.021115:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574102.021118:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574102.021119:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574102.021120:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5852 00000100:00080000:2.0:1788574102.021122:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574102.021124:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574102.021126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574102.021131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864146816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574102.021188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574102/real 1788574102] req@ffff92fc4617ce00 x1875449864146304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574102.021205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864146304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574102.021214:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574102.021218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574102.021222:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864146304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574102.021247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574102.021273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574102.021288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574102/real 1788574102] req@ffff92fc4617d500 x1875449864146432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574102.021330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864146432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574102.021346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574102.021349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574102.021352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574102.021357:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574102.021471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574102.021530:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574102/real 1788574102] req@ffff92fc4617ea00 x1875449864146560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574102.021550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864146560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574102.021558:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574102.021562:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574102.021565:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574102.021571:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574102.021594:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574102.021605:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574102/real 1788574102] req@ffff92fc4617e300 x1875449864146688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574102.021617:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864146688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574102.021624:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574102.021627:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574102.021630:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574102.021637:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574102.021657:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574102.021669:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574102/real 1788574102] req@ffff92fc4617d880 x1875449864146816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574102.021682:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864146816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574102.021688:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574102.021690:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574102.021693:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864146816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574102.468834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574102.532829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574103.492849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574103.556799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574104.516865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574104.580829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574105.540882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574105.604785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574106.564868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574106.628827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574107.140849:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574107.140856:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574107.140860:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574107.140873:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5857 00000100:00080000:2.0:1788574107.140882:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574107.140887:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574107.140925:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864146944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574107.140945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574107.140949:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574107.140950:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574107.140953:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5857 00000100:00080000:2.0:1788574107.140955:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574107.140958:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574107.140966:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574107.140975:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864147072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574107.140983:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574107.140986:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574107.140989:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574107.140991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864146944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574107.140994:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5857 00000100:00080000:2.0:1788574107.141000:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574107.141003:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574107.141006:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574107.141017:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864147200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574107.141028:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574107.141032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574107.141035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574107.141039:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5857 00000100:00080000:2.0:1788574107.141051:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574107.141056:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574107.141062:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574107.141062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574107.141076:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864147328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574107.141088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574107.141091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574107.141092:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574107.141095:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5857 00000100:00080000:2.0:1788574107.141099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574107.141102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574107.141105:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574107.141109:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574107/real 1788574107] req@ffff92fc4617c000 x1875449864146944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574107.141117:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864147456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574107.141131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864146944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574107.141141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574107.141144:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574107.141148:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864146944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574107.141185:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574107.141203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574107.141212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574107/real 1788574107] req@ffff92fc4617fb80 x1875449864147072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574107.141220:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864147072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574107.141230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574107.141233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574107.141235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574107.141239:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574107.141257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574107.141268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574107/real 1788574107] req@ffff92fc4617c380 x1875449864147200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574107.141282:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864147200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574107.141291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574107.141295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574107.141299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574107.141304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574107.141319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574107.141327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574107/real 1788574107] req@ffff92fc4617d180 x1875449864147328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574107.141335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864147328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574107.141340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574107.141342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574107.141345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574107.141349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574107.141359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574107.141365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574107/real 1788574107] req@ffff92fc4617e680 x1875449864147456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574107.141372:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864147456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574107.141377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574107.141379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574107.141381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574107.588859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574107.652784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574108.612854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574108.676843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574109.636899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574109.700807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574110.660868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574110.724917:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574111.684922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574111.748842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574112.260809:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574112.260819:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574112.260823:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574112.260834:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5862 00000100:00080000:2.0:1788574112.260844:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574112.260850:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574112.260881:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864147584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574112.260905:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574112.260908:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574112.260910:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574112.260912:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5862 00000100:00080000:2.0:1788574112.260915:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574112.260917:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574112.260926:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574112.260935:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864147712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574112.260951:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864147584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574112.260953:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574112.260959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574112.260960:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574112.260964:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5862 00000100:00080000:2.0:1788574112.260966:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574112.260968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574112.260972:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574112.260980:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864147840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574112.260987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574112.260990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574112.260991:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574112.261018:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5862 00000100:00080000:2.0:1788574112.261022:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574112.261025:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574112.261029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574112.261036:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574112.261040:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864147968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574112.261049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574112.261051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574112.261053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574112.261056:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5862 00000100:00080000:2.0:1788574112.261059:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574112.261062:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574112.261065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574112.261073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864148096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574112.261085:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff92fc4617dc00 x1875449864147584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574112.261103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864147584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574112.261111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574112.261114:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574112.261117:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864147584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574112.261136:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574112.261153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574112.261162:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff92fc4617ca80 x1875449864147712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574112.261170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864147712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574112.261176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574112.261178:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574112.261181:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574112.261184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574112.261194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574112.261201:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff92fc4617c700 x1875449864147840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574112.261208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864147840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574112.261213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574112.261214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574112.261216:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574112.261220:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574112.261230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574112.261236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff92fc4617f800 x1875449864147968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574112.261243:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864147968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574112.261248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574112.261250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574112.261252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864147968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574112.261256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574112.261265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574112.261271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574112/real 1788574112] req@ffff92fc4617f100 x1875449864148096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574112.261277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864148096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574112.261282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574112.261284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574112.261286:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574112.708839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574112.772847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574113.732831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574113.796801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574114.756823:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574114.820847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574115.780862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574115.844818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574116.804901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574116.868852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574117.380881:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.380893:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574117.380897:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.380908:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5867 00000100:00080000:3.0:1788574117.380919:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.380926:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574117.380980:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864148224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574117.381004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.381011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574117.381012:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.381015:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5867 00000100:00080000:3.0:1788574117.381019:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.381022:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574117.381031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574117.381042:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864148352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574117.381046:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864148224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574117.381057:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.381060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574117.381061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.381064:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5867 00000100:00080000:3.0:1788574117.381067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.381069:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574117.381072:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574117.381081:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864148480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574117.381088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.381090:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574117.381092:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.381094:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5867 00000100:00080000:3.0:1788574117.381096:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.381099:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574117.381103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574117.381113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574117.381113:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864148608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574117.381123:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574117.381126:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574117.381128:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574117.381131:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5867 00000100:00080000:3.0:1788574117.381133:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574117.381136:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574117.381139:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574117.381152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864148736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574117.381157:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff92fc6733bb80 x1875449864148224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.381180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864148224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.381192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.381197:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574117.381201:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864148224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574117.381223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574117.381242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574117.381251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff92fc6733b480 x1875449864148352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.381259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864148352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.381265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.381267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574117.381269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574117.381273:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574117.381284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574117.381289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff92fc6733aa00 x1875449864148480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.381297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864148480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.381302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.381304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574117.381306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574117.381309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574117.381320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574117.381326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff92fc67338000 x1875449864148608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.381334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864148608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.381339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.381341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574117.381343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574117.381346:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574117.381356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574117.381362:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574117/real 1788574117] req@ffff92fc67338700 x1875449864148736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574117.381369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864148736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574117.381374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574117.381376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574117.381378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574117.828924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574117.892898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574118.852830:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574118.916828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574119.876876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574119.940863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574120.900898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574120.964832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574121.924837:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574121.988820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574122.500792:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.500800:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574122.500803:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.500812:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5873 00000100:00080000:3.0:1788574122.500828:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.500835:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574122.500866:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864148864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.500886:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.500890:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574122.500891:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.500893:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5873 00000100:00080000:3.0:1788574122.500896:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.500899:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574122.500907:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574122.500915:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864148992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.500923:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.500925:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574122.500926:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.500929:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5873 00000100:00080000:3.0:1788574122.500931:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.500933:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574122.500936:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574122.500941:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864148864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574122.500945:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864149120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.500956:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.500958:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574122.500960:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.500962:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5873 00000100:00080000:3.0:1788574122.500964:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.500966:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574122.500969:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574122.500977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864149248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574122.500989:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574122.500991:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574122.500992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574122.500994:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5873 00000100:00080000:3.0:1788574122.500997:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574122.500998:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574122.501002:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574122.501009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864149376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574122.501018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574122.501062:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff92fc6733a300 x1875449864148864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.501080:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864148864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.501098:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.501102:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574122.501106:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864148864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574122.501113:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574122.501133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574122.501145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff92fc67338e00 x1875449864148992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.501157:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864148992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.501167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.501170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574122.501173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864148992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574122.501197:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574122.501215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574122.501225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff92fc6733a680 x1875449864149120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.501236:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864149120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.501244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.501248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574122.501251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574122.501258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574122.501275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574122.501284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff92fc67338700 x1875449864149248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.501296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864149248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.501304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.501307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574122.501310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574122.501316:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574122.501331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574122.501341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574122/real 1788574122] req@ffff92fc67338000 x1875449864149376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574122.501352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864149376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574122.501359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574122.501363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574122.501366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574122.948852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574123.012804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574123.972849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574124.036823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574124.996883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574125.060816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574126.020898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574126.084821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574127.044864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574127.108804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574127.620802:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574127.620811:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574127.620814:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574127.620823:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5878 00000100:00080000:3.0:1788574127.620832:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574127.620839:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574127.620872:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864149504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574127.620895:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574127.620898:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574127.620899:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574127.620902:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5878 00000100:00080000:3.0:1788574127.620904:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574127.620906:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574127.620915:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574127.620923:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864149632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574127.620931:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574127.620933:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574127.620935:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574127.620937:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5878 00000100:00080000:3.0:1788574127.620939:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574127.620942:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574127.620945:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574127.620953:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864149760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574127.620955:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864149504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574127.620963:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574127.620967:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574127.620968:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574127.620971:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5878 00000100:00080000:3.0:1788574127.620973:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574127.620975:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574127.620979:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574127.620986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864149888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574127.620992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574127.620994:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574127.620996:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574127.620998:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5878 00000100:00080000:3.0:1788574127.621001:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574127.621004:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574127.621007:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574127.621016:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864150016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574127.621064:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574127.621107:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574127.621131:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574127.621152:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574128.068850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574128.132812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574129.092858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574129.156787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574130.116796:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574130.180815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574130.693160:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574130.693207:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574130.693225:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574130.693257:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574130.693273:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788574130.693275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff92fc6733aa00 x1875449864149504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788574130.693285:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574130.693299:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788574130.693314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864149504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574130.693329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574130.693335:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574130.693339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864149504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574130.693349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff92fc6733b480 x1875449864149632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574130.693366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864149632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574130.693377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574130.693381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574130.693386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574130.693415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff92fc6733bb80 x1875449864149760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574130.693455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864149760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574130.693467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574130.693472:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574130.693476:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574130.693486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff92fc6733b100 x1875449864149888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574130.693507:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864149888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574130.693516:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574130.693520:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574130.693524:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864149888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574130.693532:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574130] req@ffff92fc67339500 x1875449864150016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574130.693544:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864150016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574130.693553:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574130.693557:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574130.693560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574131.140888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574131.204778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574132.164841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574132.228803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574132.740787:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.740796:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574132.740800:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.740809:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5883 00000100:00080000:3.0:1788574132.740818:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.740825:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574132.740856:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864150144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.740882:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.740886:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574132.740887:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.740890:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5883 00000100:00080000:3.0:1788574132.740893:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.740895:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574132.740902:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574132.740912:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864150272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.740923:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.740926:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574132.740935:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.740938:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5883 00000100:00080000:3.0:1788574132.740940:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.740943:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574132.740948:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574132.740950:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864150144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574132.740963:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864150400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.740976:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.740979:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574132.740982:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574132.740995:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5883 00000100:00080000:3.0:1788574132.741000:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.741004:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574132.741010:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574132.741022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864150528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574132.741032:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574132.741035:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574132.741037:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574132.741043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574132.741044:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5883 00000100:00080000:3.0:1788574132.741051:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574132.741057:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574132.741063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574132.741075:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864150656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574132.741095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff92fc67338380 x1875449864150144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.741118:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864150144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.741130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.741135:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574132.741139:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864150144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574132.741148:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574132.741177:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574132.741191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff92fc67338a80 x1875449864150272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.741203:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864150272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.741212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.741215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574132.741219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574132.741245:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574132.741266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574132.741277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff92fc6733b800 x1875449864150400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.741288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864150400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.741303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.741306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574132.741309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574132.741315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574132.741335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574132.741347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff92fc67339880 x1875449864150528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.741360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864150528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.741404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.741408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574132.741412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574132.741419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574132.741441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574132.741453:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff92fc67339c00 x1875449864150656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.741466:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864150656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.741474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.741477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574132.741482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574133.188877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574133.252823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574134.212839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574134.276802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574135.236864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574135.300773:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574136.260811:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574136.324790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574137.284833:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574137.348879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574137.860824:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.860833:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574137.860837:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.860846:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5888 00000100:00080000:0.0:1788574137.860859:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.860866:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574137.860902:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864150784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.860926:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.860930:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574137.860931:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.860934:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5888 00000100:00080000:0.0:1788574137.860937:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.860941:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574137.860949:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574137.860958:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864150912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.860968:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.860971:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574137.860972:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864150784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574137.860973:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.860981:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5888 00000100:00080000:0.0:1788574137.860984:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.860987:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574137.860991:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574137.861002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864151040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.861009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.861013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574137.861016:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.861020:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5888 00000100:00080000:0.0:1788574137.861026:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.861031:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574137.861037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574137.861048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574137.861052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864151168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574137.861063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574137.861066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574137.861067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574137.861071:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5888 00000100:00080000:0.0:1788574137.861122:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574137.861128:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574137.861133:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574137.861148:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff92fc6704bb80 x1875449864150784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574137.861149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864151296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574137.861300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864150784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574137.861310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574137.861314:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574137.861318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864150784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574137.861409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574137.861430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574137.861441:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff92fc67048e00 x1875449864150912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574137.861450:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864150912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574137.861455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574137.861458:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574137.861461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864150912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574137.861465:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574137.861476:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574137.861483:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff92fc67049c00 x1875449864151040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574137.861491:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864151040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574137.861496:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574137.861498:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574137.861500:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574137.861503:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574137.861514:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574137.861520:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff92fc67049180 x1875449864151168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574137.861527:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864151168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574137.861532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574137.861534:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574137.861536:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574137.861539:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574137.861548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574137.861554:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff92fc67049500 x1875449864151296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574137.861561:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864151296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574137.861566:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574137.861568:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574137.861570:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574138.308937:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574138.372808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574139.332865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574139.396820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574140.356788:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574140.420804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574141.380863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574141.444822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574142.404817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574142.468810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574142.980822:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.980832:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574142.980837:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.980847:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5893 00000100:00080000:0.0:1788574142.980858:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.980866:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574142.980901:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864151424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574142.980923:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.980926:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574142.980927:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.980930:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5893 00000100:00080000:0.0:1788574142.980933:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.980936:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574142.980945:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574142.980959:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864151552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574142.980968:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864151424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574142.980973:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.980977:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574142.980979:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.981018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5893 00000100:00080000:0.0:1788574142.981022:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.981027:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574142.981036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574142.981052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864151680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574142.981084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.981088:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574142.981090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574142.981095:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5893 00000800:00000100:3.0:1788574142.981098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574142.981102:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574142.981109:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574142.981115:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574142.981130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864151808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574142.981139:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff92fc6704b100 x1875449864151424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574142.981142:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574142.981153:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574142.981156:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574142.981160:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864151424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574142.981164:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5893 00000100:00080000:0.0:1788574142.981173:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574142.981174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574142.981180:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:3.0:1788574142.981181:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574142.981187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864151424:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788574142.981188:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574142.981203:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864151936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574142.981224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574142.981243:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574142.981253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff92fc67049880 x1875449864151552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574142.981261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864151552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574142.981271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574142.981274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574142.981276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574142.981280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574142.981290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574142.981297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff92fc6704ad80 x1875449864151680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574142.981304:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864151680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574142.981309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574142.981311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574142.981313:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574142.981316:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574142.981327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574142.981333:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff92fc6704a680 x1875449864151808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574142.981340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864151808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574142.981344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574142.981346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574142.981349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574142.981352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574142.981362:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574142.981368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574142] req@ffff92fc6704aa00 x1875449864151936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574142.981374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864151936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574142.981379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574142.981381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574142.981383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864151936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574143.428882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574143.492801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574144.452807:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574144.516809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574145.476866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574145.540819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574146.500875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574146.564803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574147.524850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574147.588851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574148.100889:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574148.100899:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574148.100919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574148.100930:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5898 00000100:00080000:2.0:1788574148.100941:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574148.100950:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574148.100988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864152064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574148.101020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574148.101025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574148.101028:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574148.101032:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5898 00000100:00080000:2.0:1788574148.101036:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574148.101040:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574148.101051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574148.101065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864152192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574148.101077:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574148.101079:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864152064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574148.101085:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574148.101089:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574148.101093:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5898 00000100:00080000:2.0:1788574148.101096:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574148.101100:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574148.101106:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574148.101119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864152320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574148.101135:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574148.101139:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574148.101141:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574148.101144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574148.101148:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5898 00000100:00080000:2.0:1788574148.101154:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574148.101158:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574148.101164:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574148.101177:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864152448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574148.101188:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574148.101191:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574148.101193:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574148.101197:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5898 00000100:00080000:2.0:1788574148.101201:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574148.101206:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574148/real 1788574148] req@ffff92fc4617ed80 x1875449864152064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574148.101208:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574148.101216:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574148.101234:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864152576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574148.101239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864152064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574148.101262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574148.101266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574148.101269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864152064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574148.101291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574148.101312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574148.101353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574148/real 1788574148] req@ffff92fc4617f480 x1875449864152192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574148.101367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864152192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574148.101377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574148.101381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574148.101384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574148.101397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574148.101418:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574148.101432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574148/real 1788574148] req@ffff92fc4617f100 x1875449864152320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574148.101460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864152320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574148.101469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574148.101473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574148.101476:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574148.101483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574148.101503:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574148.101514:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574148/real 1788574148] req@ffff92fc4617f800 x1875449864152448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574148.101526:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864152448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574148.101535:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574148.101538:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574148.101542:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574148.101546:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574148.101560:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574148.101575:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574148/real 1788574148] req@ffff92fc4617c700 x1875449864152576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574148.101583:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864152576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574148.101588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574148.101591:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574148.101593:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574148.548852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574148.612832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574149.572838:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574149.636820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574150.596854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574150.660819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574151.620881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574151.684887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574152.644879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574152.708785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574153.220864:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574153.220875:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574153.220879:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574153.220889:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5903 00000100:00080000:3.0:1788574153.220899:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574153.220906:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574153.220937:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864152704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574153.220962:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574153.220967:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574153.220969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574153.220974:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5903 00000100:00080000:3.0:1788574153.220978:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574153.220983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574153.220993:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574153.221003:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864152704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574153.221007:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864152832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574153.221017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574153.221021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574153.221022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574153.221025:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5903 00000100:00080000:3.0:1788574153.221028:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574153.221030:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574153.221034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574153.221042:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864152960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574153.221049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574153.221052:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574153.221053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574153.221055:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5903 00000100:00080000:3.0:1788574153.221057:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574153.221060:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574153.221065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574153.221070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574153.221075:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864153088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574153.221083:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574153.221085:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574153.221087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574153.221089:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5903 00000100:00080000:3.0:1788574153.221091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574153.221094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574153.221097:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574153.221105:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864153216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574153.221110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff92fc67339180 x1875449864152704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574153.221127:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864152704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574153.221136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574153.221139:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574153.221142:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864152704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574153.221161:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574153.221175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574153.221184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff92fc67339f80 x1875449864152832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574153.221191:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864152832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574153.221197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574153.221199:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574153.221202:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574153.221205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574153.221215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574153.221222:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff92fc6733ad80 x1875449864152960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574153.221229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864152960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574153.221235:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574153.221237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574153.221239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864152960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574153.221242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574153.221252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574153.221258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff92fc67339c00 x1875449864153088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574153.221264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864153088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574153.221269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574153.221278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574153.221280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574153.221284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574153.221295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574153.221302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574153/real 1788574153] req@ffff92fc67339880 x1875449864153216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574153.221309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864153216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574153.221314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574153.221316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574153.221318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574153.668894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574153.732886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574154.692919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574154.756876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574155.716855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574155.780852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574156.740835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574156.804835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574157.764805:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574157.828807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574158.340818:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574158.340829:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574158.340834:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574158.340844:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5908 00000100:00080000:3.0:1788574158.340854:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574158.340862:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574158.340898:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864153344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574158.340926:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574158.340931:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574158.340933:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574158.340937:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5908 00000100:00080000:3.0:1788574158.340942:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574158.340948:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574158.340958:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574158.340966:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864153344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574158.340974:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864153472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574158.340987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574158.340991:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574158.340994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574158.340999:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5908 00000100:00080000:3.0:1788574158.341004:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574158.341009:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574158.341014:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574158.341025:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574158.341029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864153600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574158.341042:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574158.341046:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574158.341048:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574158.341053:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5908 00000100:00080000:3.0:1788574158.341058:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574158.341062:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574158.341066:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff92fc6733b800 x1875449864153344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574158.341070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574158.341094:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864153344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574158.341094:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864153728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574158.341116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574158.341124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574158.341131:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864153344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574158.341136:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574158.341141:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574158.341143:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574158.341149:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5908 00000100:00080000:3.0:1788574158.341154:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574158.341158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153472:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788574158.341162:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574158.341168:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574158.341183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864153856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574158.341192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574158.341207:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff92fc67338a80 x1875449864153472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574158.341217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864153472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574158.341224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574158.341227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574158.341229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574158.341234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574158.341245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574158.341252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff92fc67338380 x1875449864153600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574158.341267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864153600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574158.341272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574158.341275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574158.341277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574158.341281:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574158.341292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574158.341298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff92fc67339500 x1875449864153728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574158.341305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864153728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574158.341311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574158.341313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574158.341315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574158.341329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574158.341349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574158.341361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574158/real 1788574158] req@ffff92fc6733b100 x1875449864153856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574158.341374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864153856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574158.341380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574158.341383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574158.341385:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864153856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574158.788830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574158.852836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574159.812824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574159.876796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574160.836849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574160.900852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574161.860891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574161.924803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574162.884846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574162.948832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574163.460885:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.460905:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574163.460911:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.460925:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5914 00000100:00080000:3.0:1788574163.460940:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.460950:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574163.461001:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864153984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574163.461039:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.461044:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574163.461046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.461051:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5914 00000100:00080000:3.0:1788574163.461055:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.461059:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574163.461074:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574163.461092:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864154112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574163.461096:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864153984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574163.461114:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.461118:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574163.461121:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.461127:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5914 00000100:00080000:3.0:1788574163.461131:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.461136:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574163.461142:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574163.461167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864154240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574163.461174:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574163.461184:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.461189:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574163.461192:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574163.461197:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5914 00000100:00080000:3.0:1788574163.461202:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.461207:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574163.461214:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574163.461227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864154368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574163.461237:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574163.461241:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574163.461244:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574163.461250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff92fc6733bb80 x1875449864153984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574163.461251:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5914 00000100:00080000:3.0:1788574163.461258:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574163.461267:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574163.461274:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574163.461288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864153984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574163.461291:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864154496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574163.461313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.461319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574163.461324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864153984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574163.461358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574163.461387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574163.461403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff92fc6733b480 x1875449864154112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.461419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864154112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.461453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.461459:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574163.461464:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574163.461472:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574163.461512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574163.461528:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff92fc6733aa00 x1875449864154240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.461542:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864154240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.461554:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.461559:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574163.461565:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574163.461584:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574163.461612:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574163.461626:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff92fc67338000 x1875449864154368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.461640:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864154368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.461659:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.461664:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574163.461668:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574163.461675:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574163.461699:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574163.461717:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574163/real 1788574163] req@ffff92fc67338700 x1875449864154496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574163.461752:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864154496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574163.461758:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574163.461760:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574163.461763:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574163.908911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574163.972866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574164.932829:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574164.996834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574165.956859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574166.020834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574166.980847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574167.044843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574168.004880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574168.068809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574168.580826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.580834:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574168.580838:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.580847:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5919 00000100:00080000:2.0:1788574168.580856:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.580863:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574168.580894:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864154624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.580914:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.580917:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574168.580919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.580921:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5919 00000100:00080000:2.0:1788574168.580923:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.580927:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574168.580934:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574168.580942:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864154752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.580949:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.580952:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574168.580953:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.580956:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5919 00000100:00080000:2.0:1788574168.580959:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.580961:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574168.580965:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574168.580971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864154880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.580978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.580980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574168.580982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.580988:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5919 00000100:00100000:1.0:1788574168.580996:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864154624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574168.580996:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.581011:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574168.581019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574168.581041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864155008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.581049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.581052:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574168.581053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.581057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5919 00000100:00080000:2.0:1788574168.581062:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.581064:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574168.581069:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574168.581077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864155136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574168.581087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574168.581137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff92fc4617ca80 x1875449864154624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.581155:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864154624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.581165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.581168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574168.581172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864154624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574168.581186:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574168.581207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574168.581217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff92fc4617dc00 x1875449864154752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.581225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864154752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.581232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.581234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574168.581237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574168.581308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574168.581329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574168.581340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff92fc4617e680 x1875449864154880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.581352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864154880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.581359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.581361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574168.581363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864154880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574168.581368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574168.581384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574168.581391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff92fc4617d180 x1875449864155008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.581399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864155008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.581405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.581407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574168.581409:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574168.581413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574168.581423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574168.581429:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff92fc4617c380 x1875449864155136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574168.581436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864155136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574168.581442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574168.581444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574168.581446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574169.028909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574169.092877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574170.052907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574170.116875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574171.076865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574171.140851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574172.100940:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574172.164873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574173.124835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574173.188880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574173.700864:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.700881:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574173.700887:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.700911:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5924 00000100:00080000:2.0:1788574173.700926:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.700937:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574173.701018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864155264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.701056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.701060:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574173.701063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.701067:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5924 00000100:00080000:2.0:1788574173.701072:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.701077:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574173.701091:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574173.701109:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864155392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574173.701119:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864155264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574173.701126:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.701131:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574173.701133:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.701139:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5924 00000100:00080000:2.0:1788574173.701144:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.701150:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574173.701156:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574173.701171:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864155520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.701181:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.701185:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574173.701188:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.701193:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5924 00000100:00080000:2.0:1788574173.701198:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.701203:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574173.701211:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574173.701224:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864155648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.701234:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.701238:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574173.701240:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.701244:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5924 00000100:00080000:2.0:1788574173.701249:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.701253:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574173.701260:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574173.701272:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864155776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574173.701288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574173.701354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff92fc4617fb80 x1875449864155264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574173.701377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864155264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574173.701390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574173.701395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574173.701399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864155264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574173.701574:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574173.701668:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574173.701684:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff92fc4617c000 x1875449864155392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574173.701698:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864155392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574173.701714:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574173.701718:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574173.701753:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574173.701762:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574173.701787:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574173.701799:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff92fc4617d880 x1875449864155520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574173.701811:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864155520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574173.701820:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574173.701824:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574173.701828:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574173.701834:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574173.701854:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574173.701864:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff92fc4617e300 x1875449864155648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574173.701877:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864155648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574173.701885:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574173.701888:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574173.701892:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574173.701898:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574173.701915:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574173.701925:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff92fc4617ea00 x1875449864155776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574173.701937:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864155776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574173.701946:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574173.701965:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574173.701970:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864155776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574174.148975:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574174.212883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574175.172888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574175.236895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574176.196943:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574176.260881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574177.220814:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574177.284837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574178.244909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574178.308786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574178.820846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.820856:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574178.820861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.820874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5929 00000100:00080000:3.0:1788574178.820886:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.820895:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574178.820946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864155904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574178.820984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.820988:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574178.820990:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.820994:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5929 00000100:00080000:3.0:1788574178.820997:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.821001:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574178.821013:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574178.821030:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864156032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574178.821038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864155904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574178.821049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.821053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574178.821056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.821061:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5929 00000100:00080000:3.0:1788574178.821065:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.821068:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574178.821073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574178.821085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864156160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574178.821093:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.821095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574178.821097:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.821099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5929 00000100:00080000:3.0:1788574178.821102:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.821104:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574178.821107:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574178.821120:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864156288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574178.821124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574178.821132:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.821137:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574178.821138:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.821142:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5929 00000100:00080000:3.0:1788574178.821145:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.821148:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574178.821151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574178.821159:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864156416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574178.821180:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff92fc6733a680 x1875449864155904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.821198:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864155904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.821207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.821211:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574178.821214:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864155904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574178.821238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574178.821260:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574178.821270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff92fc67338e00 x1875449864156032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.821278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864156032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.821284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.821286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574178.821289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574178.821292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574178.821304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574178.821311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff92fc6733a300 x1875449864156160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.821318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864156160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.821323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.821325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574178.821327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574178.821331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574178.821343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574178.821349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff92fc67338700 x1875449864156288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.821357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864156288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.821363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.821368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574178.821370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574178.821374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574178.821385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574178.821391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff92fc67338000 x1875449864156416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574178.821399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864156416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574178.821405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574178.821407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574178.821409:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574179.268945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574179.332864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574180.292913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574180.356871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574181.316857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574181.380820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574182.340839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574182.404836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574183.364838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574183.428815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574183.940839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574183.940847:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574183.940850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574183.940859:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5934 00000100:00080000:0.0:1788574183.940868:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574183.940874:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574183.940907:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864156544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574183.940928:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574183.940931:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574183.940933:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574183.940935:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5934 00000100:00080000:0.0:1788574183.940938:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574183.940940:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574183.940947:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574183.940956:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864156672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574183.940966:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574183.940969:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574183.940970:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574183.940976:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5934 00000100:00100000:1.0:1788574183.940976:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864156544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574183.940980:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574183.940985:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574183.940989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574183.940996:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864156800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574183.941002:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574183.941004:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574183.941005:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574183.941018:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5934 00000100:00080000:0.0:1788574183.941033:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574183.941037:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574183.941042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574183.941044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574183.941059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864156928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574183.941070:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574183.941073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574183.941075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574183.941080:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5934 00000100:00080000:0.0:1788574183.941085:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574183.941087:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff92fc6704a300 x1875449864156544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574183.941092:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574183.941105:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574183.941111:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864156544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574183.941121:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864157056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574183.941157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.941163:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574183.941168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864156544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574183.941190:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574183.941208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574183.941220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff92fc67048000 x1875449864156672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.941233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864156672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.941242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.941245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574183.941249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574183.941254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574183.941268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574183.941277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff92fc67049f80 x1875449864156800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.941289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864156800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.941297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.941300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574183.941303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574183.941309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574183.941324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574183.941333:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff92fc67048700 x1875449864156928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.941344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864156928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.941353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.941372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574183.941377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864156928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574183.941383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574183.941403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574183.941414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff92fc67048380 x1875449864157056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574183.941427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864157056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574183.941435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574183.941439:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574183.941442:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574184.388847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574184.452866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574185.412921:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574185.476816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574186.436908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574186.500888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574187.460867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574187.524786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574188.484884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574188.548846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574189.060885:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574189.060902:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574189.060908:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574189.060920:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5939 00000100:00080000:3.0:1788574189.060933:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574189.060941:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574189.060996:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864157184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574189.061032:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574189.061037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574189.061039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574189.061044:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5939 00000100:00080000:3.0:1788574189.061049:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574189.061054:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574189.061082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574189.061090:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864157184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574189.061103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864157312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574189.061118:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574189.061122:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574189.061124:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574189.061128:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5939 00000100:00080000:3.0:1788574189.061132:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574189.061136:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574189.061141:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574189.061151:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864157440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574189.061158:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574189.061161:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574189.061162:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574189.061166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574189.061167:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5939 00000100:00080000:3.0:1788574189.061173:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574189.061176:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574189.061181:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574189.061190:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864157568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574189.061197:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574189.061199:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574189.061200:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574189.061203:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5939 00000100:00080000:3.0:1788574189.061206:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574189.061210:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574189.061213:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574189.061220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574189/real 1788574189] req@ffff92fc6733aa00 x1875449864157184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574189.061227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864157696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574189.061245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864157184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574189.061255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574189.061259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574189.061263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864157184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574189.061287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574189.061302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574189.061311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574189/real 1788574189] req@ffff92fc6733b480 x1875449864157312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574189.061320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864157312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574189.061326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574189.061328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574189.061330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574189.061333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574189.061345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574189.061351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574189/real 1788574189] req@ffff92fc6733bb80 x1875449864157440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574189.061359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864157440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574189.061369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574189.061372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574189.061374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574189.061377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574189.061389:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574189.061394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574189/real 1788574189] req@ffff92fc6733b100 x1875449864157568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574189.061401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864157568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574189.061406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574189.061408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574189.061411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574189.061414:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574189.061424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574189.061429:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574189/real 1788574189] req@ffff92fc67339500 x1875449864157696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574189.061436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864157696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574189.061440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574189.061443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574189.061446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574189.508859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574189.572784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574190.532857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574190.596833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574191.556990:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574191.620869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574192.580875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574192.644887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574193.604906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574193.668888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574194.180839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574194.180857:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574194.180863:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574194.180873:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5944 00000100:00080000:1.0:1788574194.180885:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574194.180893:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574194.180938:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864157824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574194.180969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574194.180973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574194.180976:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574194.180981:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5944 00000100:00080000:1.0:1788574194.180985:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574194.180991:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574194.181003:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574194.181021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864157952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574194.181027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864157824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574194.181037:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574194.181040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574194.181042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574194.181047:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5944 00000100:00080000:1.0:1788574194.181053:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574194.181060:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574194.181068:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574194.181086:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864158080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574194.181098:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574194.181104:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574194.181107:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574194.181112:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5944 00000100:00080000:1.0:1788574194.181118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574194.181125:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574194.181132:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574194.181140:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574194.181151:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864158208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574194.181169:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574194.181174:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574194.181177:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574194.181182:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158080:192.168.202.132@tcp:38:kworker.0 00000100:00080000:1.0:1788574194.181184:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5944 00000100:00080000:1.0:1788574194.181194:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574194.181201:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574194.181208:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574194.181214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574194.181236:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864158336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574194.181276:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574194.628899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574194.692853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574195.652943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574195.716899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574196.676819:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574196.740823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574197.253033:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574197.253080:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574197.253098:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574197.253141:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574197.253152:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574197.253159:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574197.253168:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574197.253288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff92fc6728ad80 x1875449864158336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.253323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864158336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.253337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.253356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574197.253362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158336:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574197.253371:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff92fc67289880 x1875449864158208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.253384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864158208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.253394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.253399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574197.253402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574197.253409:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff92fc67289f80 x1875449864157952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.253422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864157952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.253431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.253434:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574197.253437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864157952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574197.253445:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff92fc67288000 x1875449864158080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.253456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864158080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.253464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.253467:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574197.253471:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574197.253478:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574194/real 1788574197] req@ffff92fc67288e00 x1875449864157824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574197.253490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864157824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574197.253500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574197.253504:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574197.253508:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864157824:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788574197.700882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574197.764845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574198.724805:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574198.788842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574199.300823:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574199.300831:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574199.300834:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574199.300843:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5949 00000100:00080000:3.0:1788574199.300852:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574199.300858:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574199.300893:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864158464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574199.300915:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574199.300918:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574199.300920:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574199.300922:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5949 00000100:00080000:3.0:1788574199.300925:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574199.300928:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574199.300936:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574199.300945:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864158592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574199.300954:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864158464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574199.300954:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574199.300960:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574199.300964:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574199.300968:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5949 00000100:00080000:3.0:1788574199.300970:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574199.300974:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574199.300979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574199.300992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864158720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574199.301002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574199.301006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574199.301008:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574199.301012:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5949 00000100:00080000:3.0:1788574199.301017:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574199.301021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574199.301026:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574199.301030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574199.301042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864158848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574199.301053:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574199.301057:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574199.301060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574199.301064:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5949 00000100:00080000:3.0:1788574199.301067:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574199.301071:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574199.301079:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff92fc67338380 x1875449864158464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574199.301082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574199.301100:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864158976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574199.301110:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864158464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574199.301125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574199.301130:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574199.301134:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864158464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574199.301160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574199.301188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574199.301203:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff92fc67338a80 x1875449864158592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574199.301216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864158592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574199.301226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574199.301229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574199.301233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574199.301240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574199.301262:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574199.301281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff92fc6733b800 x1875449864158720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574199.301300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864158720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574199.301308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574199.301312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574199.301315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574199.301321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574199.301342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574199.301354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff92fc67339880 x1875449864158848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574199.301365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864158848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574199.301372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574199.301375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574199.301378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574199.301384:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574199.301403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574199.301414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574199/real 1788574199] req@ffff92fc67339c00 x1875449864158976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574199.301426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864158976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574199.301440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574199.301443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574199.301446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864158976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574199.748852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574199.812825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574200.772957:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574200.836839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574201.796957:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574201.860907:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574202.820875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574202.884804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574203.844875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574203.908844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574204.420824:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574204.420833:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574204.420836:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574204.420844:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5955 00000100:00080000:3.0:1788574204.420853:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574204.420860:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574204.420893:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864159104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574204.420917:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574204.420920:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574204.420922:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574204.420924:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5955 00000100:00080000:3.0:1788574204.420926:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574204.420929:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574204.420937:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574204.420946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864159232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574204.420954:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574204.420957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574204.420960:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574204.420962:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864159104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574204.420964:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5955 00000100:00080000:3.0:1788574204.420970:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574204.420974:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574204.420977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574204.420985:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864159360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574204.420992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574204.420995:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574204.420996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574204.420998:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5955 00000100:00080000:3.0:1788574204.421001:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574204.421003:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574204.421006:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574204.421017:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864159488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574204.421023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574204.421034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574204.421038:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574204.421041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574204.421046:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5955 00000100:00080000:3.0:1788574204.421051:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574204.421055:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574204.421061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574204.421070:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff92fc6733ad80 x1875449864159104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574204.421075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864159616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574204.421102:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864159104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574204.421111:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574204.421114:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574204.421119:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864159104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574204.421155:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574204.421216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574204.421238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff92fc67339f80 x1875449864159232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574204.421247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864159232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574204.421253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574204.421256:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574204.421258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574204.421263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574204.421276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574204.421283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff92fc67339180 x1875449864159360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574204.421290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864159360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574204.421296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574204.421298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574204.421300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574204.421303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574204.421314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574204.421319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff92fc67339c00 x1875449864159488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574204.421326:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864159488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574204.421332:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574204.421333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574204.421335:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574204.421339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574204.421349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574204.421355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574204/real 1788574204] req@ffff92fc67339880 x1875449864159616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574204.421362:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864159616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574204.421367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574204.421369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574204.421371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574204.868860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574204.932783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574205.892872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574205.956835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574206.916843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574206.980808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574207.940849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574208.004835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574208.964853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574209.028794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574209.540819:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574209.540827:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574209.540830:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574209.540838:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5960 00000100:00080000:3.0:1788574209.540847:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574209.540853:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574209.540897:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864159744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574209.540920:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574209.540923:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574209.540925:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574209.540930:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5960 00000100:00080000:3.0:1788574209.540934:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574209.540938:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574209.540947:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574209.540957:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864159744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574209.540962:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864159872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574209.540973:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574209.540975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574209.540976:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574209.540979:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5960 00000100:00080000:3.0:1788574209.540981:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574209.540984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574209.540988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574209.540997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864160000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574209.541005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574209.541007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574209.541009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574209.541012:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5960 00000100:00080000:3.0:1788574209.541018:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574209.541021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574209.541025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574209.541036:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574209.541037:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864160128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574209.541046:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574209.541049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574209.541050:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574209.541053:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5960 00000100:00080000:3.0:1788574209.541056:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574209.541059:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574209.541062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574209.541069:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864160256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574209.541088:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff92fc6733b800 x1875449864159744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574209.541111:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864159744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574209.541123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574209.541128:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574209.541132:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864159744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574209.541156:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574209.541181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574209.541198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff92fc67338a80 x1875449864159872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574209.541212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864159872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574209.541222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574209.541225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574209.541229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864159872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574209.541235:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574209.541254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574209.541268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff92fc67338380 x1875449864160000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574209.541281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864160000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574209.541290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574209.541294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574209.541297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574209.541304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574209.541325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574209.541337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff92fc67339500 x1875449864160128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574209.541350:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864160128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574209.541360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574209.541363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574209.541366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574209.541373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574209.541396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574209.541423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574209/real 1788574209] req@ffff92fc6733b100 x1875449864160256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574209.541438:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864160256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574209.541448:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574209.541452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574209.541456:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574209.988889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574210.052794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574211.012969:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574211.076848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574212.036843:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574212.100816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574213.060883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574213.124823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574214.084848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574214.148804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574214.660937:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.660954:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574214.660959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.660969:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5965 00000100:00080000:0.0:1788574214.660980:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.660987:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574214.661026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864160384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574214.661050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.661053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574214.661054:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.661057:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5965 00000100:00080000:0.0:1788574214.661060:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.661064:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574214.661072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574214.661084:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864160384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574214.661085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864160512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574214.661099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.661101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574214.661102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.661105:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5965 00000100:00080000:0.0:1788574214.661108:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.661111:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574214.661114:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574214.661123:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864160640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574214.661129:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.661131:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574214.661133:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.661135:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5965 00000100:00080000:0.0:1788574214.661138:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.661140:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574214.661142:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574214.661147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574214.661151:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864160768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574214.661160:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574214.661164:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574214.661166:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574214.661170:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5965 00000100:00080000:0.0:1788574214.661174:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574214.661179:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574214.661185:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574214.661185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff92fc67048a80 x1875449864160384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574214.661201:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864160896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574214.661211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864160384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.661226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.661232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574214.661239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864160384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574214.661261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574214.661278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574214.661287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff92fc67048380 x1875449864160512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.661295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864160512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.661301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.661304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574214.661306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574214.661309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574214.661320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574214.661326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff92fc67048700 x1875449864160640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.661342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864160640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.661347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.661349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574214.661351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574214.661354:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574214.661366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574214.661373:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff92fc67049f80 x1875449864160768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.661381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864160768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.661386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.661388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574214.661390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574214.661393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574214.661403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574214.661409:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff92fc67048000 x1875449864160896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.661415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864160896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.661421:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.661422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574214.661424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864160896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574215.108922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574215.172859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574216.132824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574216.196816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574217.156893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574217.220866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574218.180948:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574218.244848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574219.204887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574219.268843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574219.780882:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574219.780893:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574219.780898:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574219.780910:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5970 00000100:00080000:3.0:1788574219.780923:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574219.780931:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574219.780978:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864161024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574219.781007:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574219.781011:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574219.781012:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574219.781015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5970 00000100:00080000:3.0:1788574219.781017:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574219.781020:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574219.781029:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574219.781041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864161152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574219.781049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574219.781052:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574219.781053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574219.781056:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5970 00000100:00080000:3.0:1788574219.781058:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574219.781060:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574219.781063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574219.781070:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864161024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574219.781072:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864161280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574219.781085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574219.781088:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574219.781089:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574219.781092:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5970 00000100:00080000:3.0:1788574219.781095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574219.781098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574219.781103:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574219.781117:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864161408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574219.781151:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574219.781164:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574219.781168:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574219.781171:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574219.781178:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5970 00000100:00080000:3.0:1788574219.781185:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574219.781191:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574219.781197:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574219.781212:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864161536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574219.781240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff92fc6733bb80 x1875449864161024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.781268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864161024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.781300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.781306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574219.781311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864161024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574219.781319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574219.781344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574219.781363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff92fc6733b480 x1875449864161152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.781373:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864161152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.781380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.781382:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574219.781385:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574219.781407:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574219.781422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574219.781434:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff92fc6733aa00 x1875449864161280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.781446:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864161280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.781455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.781459:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574219.781462:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574219.781470:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574219.781494:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574219.781506:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff92fc67338000 x1875449864161408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.781520:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864161408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.781529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.781532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574219.781535:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574219.781539:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574219.781556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574219.781565:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff92fc67338700 x1875449864161536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.781572:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864161536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.781577:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.781580:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574219.781582:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574220.228854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574220.292779:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574221.252877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574221.316796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574222.276854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574222.340787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574223.300877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574223.364831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574224.324818:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574224.388799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574224.900867:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574224.900877:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574224.900882:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574224.900893:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5975 00000100:00080000:1.0:1788574224.900906:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574224.900914:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574224.900965:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864161664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574224.901019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574224.901024:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574224.901026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574224.901030:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5975 00000100:00080000:1.0:1788574224.901035:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574224.901039:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574224.901051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574224.901066:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864161792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574224.901080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574224.901083:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574224.901084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574224.901089:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5975 00000100:00080000:1.0:1788574224.901094:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574224.901094:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864161664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574224.901098:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574224.901106:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574224.901114:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864161920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574224.901120:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574224.901123:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574224.901124:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574224.901127:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5975 00000100:00080000:1.0:1788574224.901128:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574224.901131:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574224.901134:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574224.901143:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864162048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574224.901149:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574224.901151:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574224.901152:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574224.901154:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5975 00000100:00080000:1.0:1788574224.901158:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574224.901161:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574224.901164:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574224.901171:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864162176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574224.901184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574224.901235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff92fc6728aa00 x1875449864161664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.901252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864161664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.901261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.901265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574224.901268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864161664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574224.901291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574224.901307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574224.901316:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff92fc6728b480 x1875449864161792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.901324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864161792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.901330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.901332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574224.901334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574224.901337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574224.901349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574224.901355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff92fc6728a300 x1875449864161920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.901362:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864161920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.901373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.901375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574224.901377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864161920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574224.901380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574224.901391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574224.901397:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff92fc67288e00 x1875449864162048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.901404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864162048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.901409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.901411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574224.901413:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574224.901417:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574224.901427:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574224.901433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff92fc67288000 x1875449864162176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.901440:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864162176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.901445:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.901447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574224.901449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574225.348922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574225.412921:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574226.372846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574226.436797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574227.396917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574227.460865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574228.420880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574228.484825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574229.444866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574229.508841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574230.020883:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574230.020896:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574230.020901:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574230.020913:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5980 00000100:00080000:1.0:1788574230.020924:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574230.020932:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574230.020978:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864162304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574230.021007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574230.021011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574230.021012:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574230.021015:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5980 00000100:00080000:1.0:1788574230.021017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574230.021020:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574230.021030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574230.021039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864162432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574230.021048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574230.021053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574230.021056:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574230.021059:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864162304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574230.021060:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5980 00000100:00080000:1.0:1788574230.021066:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574230.021070:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574230.021074:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574230.021083:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864162560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574230.021091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574230.021093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574230.021095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574230.021098:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5980 00000100:00080000:1.0:1788574230.021101:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574230.021103:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574230.021107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574230.021114:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864162688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574230.021121:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574230.021124:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574230.021126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574230.021128:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5980 00000100:00080000:1.0:1788574230.021131:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574230.021133:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574230.021137:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574230.021141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574230.021151:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864162816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574230.021189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574230/real 1788574230] req@ffff92fc67289f80 x1875449864162304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574230.021207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864162304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574230.021216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574230.021220:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574230.021223:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864162304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574230.021248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574230.021267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574230.021281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574230/real 1788574230] req@ffff92fc67289880 x1875449864162432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574230.021295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864162432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574230.021315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574230.021318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574230.021321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574230.021326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574230.021343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574230.021352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574230/real 1788574230] req@ffff92fc6728ad80 x1875449864162560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574230.021360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864162560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574230.021366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574230.021368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574230.021371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574230.021374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574230.021386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574230.021392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574230/real 1788574230] req@ffff92fc6728b800 x1875449864162688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574230.021399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864162688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574230.021404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574230.021406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574230.021408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574230.021411:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574230.021422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574230.021428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574230/real 1788574230] req@ffff92fc6728a680 x1875449864162816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574230.021435:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864162816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574230.021441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574230.021443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574230.021445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864162816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574230.468867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574230.532826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574231.492829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574231.556825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574232.516847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574232.580819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574233.540859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574233.604837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574234.564834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574234.628828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574235.140857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574235.140868:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574235.140873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574235.140884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5985 00000100:00080000:3.0:1788574235.140896:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574235.140904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574235.140951:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864162944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574235.140980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574235.140984:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574235.140985:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574235.140988:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5985 00000100:00080000:3.0:1788574235.140990:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574235.140993:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574235.141003:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574235.141013:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864163072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574235.141024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574235.141028:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574235.141031:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574235.141035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864162944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574235.141038:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5985 00000100:00080000:3.0:1788574235.141047:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574235.141052:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574235.141056:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574235.141069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864163200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574235.141075:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574235.141078:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574235.141080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574235.141083:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5985 00000100:00080000:3.0:1788574235.141099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574235.141102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574235.141107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574235.141116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574235.141120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864163328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574235.141129:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574235.141133:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574235.141134:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574235.141137:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5985 00000100:00080000:3.0:1788574235.141141:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574235.141145:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574235.141149:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574235.141155:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864163456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574235.141188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574235/real 1788574235] req@ffff92fc6733a300 x1875449864162944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574235.141206:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864162944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574235.141215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574235.141219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574235.141228:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864162944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574235.141252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574235.141272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574235.141282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574235/real 1788574235] req@ffff92fc67338e00 x1875449864163072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574235.141290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864163072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574235.141296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574235.141299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574235.141301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574235.141305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574235.141317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574235.141324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574235/real 1788574235] req@ffff92fc6733a680 x1875449864163200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574235.141332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864163200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574235.141337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574235.141339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574235.141341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574235.141345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574235.141356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574235.141361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574235/real 1788574235] req@ffff92fc67338700 x1875449864163328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574235.141369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864163328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574235.141374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574235.141376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574235.141378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574235.141381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574235.141391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574235.141397:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574235/real 1788574235] req@ffff92fc67338000 x1875449864163456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574235.141405:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864163456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574235.141426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574235.141428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574235.141430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574235.588818:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574235.652827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574236.612856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574236.676821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574237.636823:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574237.700812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574238.660851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574238.724831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574239.684953:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574239.748874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574240.260890:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574240.260901:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574240.260906:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574240.260917:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5990 00000100:00080000:3.0:1788574240.260929:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574240.260938:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574240.260982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864163584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574240.261007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574240.261014:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574240.261017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574240.261022:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5990 00000100:00080000:3.0:1788574240.261027:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574240.261032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574240.261040:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864163584:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788574240.261046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574240.261064:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864163712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574240.261076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574240.261079:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574240.261081:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574240.261084:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5990 00000100:00080000:3.0:1788574240.261099:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574240.261103:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574240.261107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788574240.261115:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574240.261120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864163840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574240.261128:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574240.261130:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574240.261132:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574240.261134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5990 00000100:00080000:3.0:1788574240.261137:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574240.261140:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574240.261144:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574240.261152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864163968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574240.261160:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574240.261162:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574240.261164:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574240.261166:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5990 00000100:00080000:3.0:1788574240.261170:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574240.261173:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff92fc6733aa00 x1875449864163584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574240.261176:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574240.261183:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574240.261194:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864164096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574240.261198:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864163584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574240.261223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574240.261229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574240.261234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864163584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574240.261267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574240.261294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574240.261307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff92fc6733b480 x1875449864163712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574240.261316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864163712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574240.261327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574240.261330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574240.261332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574240.261337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574240.261350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574240.261357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff92fc6733bb80 x1875449864163840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574240.261364:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864163840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574240.261370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574240.261372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574240.261374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574240.261378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574240.261390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574240.261396:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff92fc6733b100 x1875449864163968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574240.261403:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864163968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574240.261408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574240.261410:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574240.261413:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864163968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574240.261417:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574240.261444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574240.261454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574240/real 1788574240] req@ffff92fc67339500 x1875449864164096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574240.261462:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864164096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574240.261467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574240.261470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574240.261472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574240.708903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574240.772866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574241.732950:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574241.796812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574242.756879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574242.820794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574243.780888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574243.844800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574244.804937:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574244.868869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574245.380841:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574245.380850:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574245.380854:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574245.380863:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 5995 00000100:00080000:3.0:1788574245.380872:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574245.380877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574245.380910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864164224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574245.380935:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574245.380938:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574245.380940:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574245.380943:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5995 00000100:00080000:3.0:1788574245.380945:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574245.380948:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574245.380963:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574245.380973:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864164352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574245.380980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574245.380983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574245.380984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574245.380986:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5995 00000100:00080000:3.0:1788574245.380988:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574245.380990:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574245.380992:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574245.380998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864164480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574245.381005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574245.381007:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574245.381009:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574245.381011:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5995 00000100:00080000:3.0:1788574245.381013:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574245.381016:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574245.381019:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574245.381025:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864164608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574245.381036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574245.381038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574245.381039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574245.381041:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 5995 00000100:00080000:3.0:1788574245.381043:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574245.381046:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574245.381050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574245.381056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864164736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574245.381070:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864164224:192.168.202.132@tcp:250:kworker.0 00000800:00000100:2.0:1788574245.381163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574245.381225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff92fc67338380 x1875449864164224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574245.381255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864164224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574245.381273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574245.381280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574245.381285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864164224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574245.381294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574245.381323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574245.381341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff92fc67338a80 x1875449864164352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574245.381371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864164352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574245.381385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574245.381392:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574245.381397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574245.381407:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574245.381431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574245.381452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff92fc6733b800 x1875449864164480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574245.381465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864164480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574245.381474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574245.381478:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574245.381481:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574245.381489:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574245.381522:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574245.381546:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff92fc67339880 x1875449864164608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574245.381555:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864164608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574245.381562:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574245.381564:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574245.381568:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574245.381603:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574245.381630:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574245.381649:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574245/real 1788574245] req@ffff92fc67339c00 x1875449864164736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574245.381660:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864164736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574245.381666:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574245.381669:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574245.381671:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574245.828858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574245.892788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574246.852811:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574246.916818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574247.876878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574247.940776:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574248.900834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574248.964848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574249.924883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574249.988862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574250.500829:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.500837:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574250.500841:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.500849:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6001 00000100:00080000:0.0:1788574250.500857:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.500863:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574250.500897:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864164864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574250.500921:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.500924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574250.500925:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.500930:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6001 00000100:00080000:0.0:1788574250.500934:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.500939:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574250.500949:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574250.500955:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864164864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574250.500967:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864164992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574250.500979:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.500982:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574250.500984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.500988:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6001 00000100:00080000:0.0:1788574250.500991:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.500993:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574250.500997:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574250.501007:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864165120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574250.501013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.501016:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574250.501018:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574250.501022:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574250.501034:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6001 00000100:00080000:0.0:1788574250.501040:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.501042:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574250.501046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574250.501055:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864165248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574250.501060:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff92fc6704a300 x1875449864164864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574250.501066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574250.501073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574250.501076:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574250.501082:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6001 00000100:00080000:0.0:1788574250.501085:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574250.501089:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574250.501092:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864164864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574250.501093:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574250.501102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864165376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574250.501117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574250.501125:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574250.501131:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864164864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574250.501159:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574250.501185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574250.501199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff92fc6704aa00 x1875449864164992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574250.501212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864164992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574250.501221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574250.501224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574250.501228:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864164992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574250.501234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574250.501253:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574250.501263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff92fc6704a680 x1875449864165120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574250.501292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864165120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574250.501301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574250.501305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574250.501309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574250.501315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574250.501335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574250.501346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff92fc6704ad80 x1875449864165248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574250.501359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864165248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574250.501367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574250.501371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574250.501374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574250.501380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574250.501397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574250.501407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574250/real 1788574250] req@ffff92fc67049880 x1875449864165376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574250.501419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864165376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574250.501428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574250.501431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574250.501434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574250.948812:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574251.012807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574251.972901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574252.036876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574252.996840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574253.060815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574254.020847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574254.084873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574255.044817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574255.108842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574255.620858:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574255.620869:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574255.620874:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574255.620882:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6006 00000100:00080000:3.0:1788574255.620891:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574255.620897:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574255.620929:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864165504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574255.620952:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574255.620956:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574255.620958:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574255.620962:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6006 00000100:00080000:3.0:1788574255.620965:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574255.620969:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574255.620981:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574255.620995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864165632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574255.621001:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864165504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574255.621005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574255.621008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574255.621010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574255.621013:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6006 00000100:00080000:3.0:1788574255.621016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574255.621019:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574255.621024:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574255.621033:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864165760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574255.621041:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574255.621044:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574255.621046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574255.621049:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6006 00000100:00080000:3.0:1788574255.621052:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574255.621055:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788574255.621058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574255.621060:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574255.621069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864165888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574255.621078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574255.621081:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574255.621083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574255.621087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6006 00000100:00080000:3.0:1788574255.621090:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574255.621093:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574255.621098:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff92fc67339180 x1875449864165504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574255.621098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574255.621107:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864166016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574255.621115:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864165504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574255.621130:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574255.621134:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574255.621139:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864165504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574255.621163:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574255.621182:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574255.621213:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff92fc67339f80 x1875449864165632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574255.621223:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864165632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574255.621229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574255.621231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574255.621233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574255.621237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574255.621251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574255.621259:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff92fc6733ad80 x1875449864165760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574255.621266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864165760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574255.621271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574255.621273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574255.621275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574255.621281:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574255.621300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574255.621311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff92fc67339c00 x1875449864165888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574255.621322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864165888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574255.621329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574255.621333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574255.621336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864165888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574255.621342:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574255.621363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574255.621374:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff92fc67339880 x1875449864166016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574255.621384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864166016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574255.621391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574255.621394:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574255.621397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574256.068867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574256.132853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574257.092876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574257.156798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574258.116857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574258.180825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574259.140861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574259.204821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574260.164932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574260.228890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574260.740865:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.740873:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574260.740876:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.740885:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6011 00000100:00080000:2.0:1788574260.740894:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.740901:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574260.740939:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864166144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.740965:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.740968:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574260.740970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.740973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6011 00000100:00080000:2.0:1788574260.740975:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.740978:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574260.740986:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574260.740999:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864166272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.741010:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574260.741016:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864166144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574260.741017:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574260.741020:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.741028:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6011 00000100:00080000:2.0:1788574260.741032:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.741035:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574260.741040:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574260.741054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864166400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.741070:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.741073:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574260.741075:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.741078:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6011 00000100:00080000:2.0:1788574260.741082:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.741086:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574260.741091:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574260.741102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864166528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574260.741130:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574260.741133:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574260.741135:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574260.741139:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6011 00000100:00080000:2.0:1788574260.741143:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574260.741148:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574260.741154:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574260.741165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864166656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574260.741169:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574260.741208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574260.741233:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574260.741263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574261.188883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574261.252793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574262.212861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574262.276833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574263.236863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574263.300779:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788574263.812986:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788574263.813027:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574263.813040:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574263.813091:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574263.813102:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574263.813114:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574263.813126:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788574263.813136:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff92fc4617ce00 x1875449864166272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574263.813166:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864166272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574263.813179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574263.813185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574263.813189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574263.813195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff92fc4617ea00 x1875449864166400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574263.813208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864166400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574263.813217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574263.813221:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574263.813225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574263.813232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff92fc4617e300 x1875449864166528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574263.813244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864166528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574263.813252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574263.813255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574263.813259:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574263.813266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff92fc4617d500 x1875449864166144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574263.813279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864166144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574263.813288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574263.813291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574263.813295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864166144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574263.813330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574263] req@ffff92fc4617d880 x1875449864166656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574263.813343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864166656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574263.813351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574263.813356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574263.813359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574264.260851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574264.324830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574265.284844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574265.348791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574265.860853:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.860861:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574265.860865:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.860873:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6016 00000100:00080000:3.0:1788574265.860882:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.860887:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574265.860928:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864166784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.860958:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.860963:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574265.860965:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.860970:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6016 00000100:00080000:3.0:1788574265.860975:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.860979:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574265.860990:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574265.861004:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864166912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.861023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574265.861026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864166784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574265.861044:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574265.861049:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.861055:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6016 00000100:00080000:3.0:1788574265.861059:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.861064:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574265.861069:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574265.861084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864167040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.861094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.861098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574265.861100:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.861104:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6016 00000100:00080000:3.0:1788574265.861109:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.861116:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574265.861118:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574265.861124:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574265.861140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864167168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.861151:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.861155:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574265.861157:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.861162:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6016 00000100:00080000:3.0:1788574265.861168:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574265.861175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff92fc6733b800 x1875449864166784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574265.861175:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574265.861184:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574265.861203:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864167296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574265.861209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864166784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574265.861230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574265.861236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574265.861242:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864166784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574265.861270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574265.861300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574265.861315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff92fc67338a80 x1875449864166912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574265.861328:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864166912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574265.861339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574265.861342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574265.861346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864166912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574265.861352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574265.861373:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574265.861385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff92fc67338380 x1875449864167040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574265.861398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864167040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574265.861405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574265.861409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574265.861412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574265.861419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574265.861439:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574265.861451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff92fc67339500 x1875449864167168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574265.861465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864167168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574265.861474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574265.861477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574265.861481:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574265.861488:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574265.861510:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574265.861523:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff92fc6733b100 x1875449864167296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574265.861543:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864167296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574265.861552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574265.861556:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574265.861560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574266.308856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574266.372797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574267.332811:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574267.396783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574268.356855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574268.420864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574269.380869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574269.444850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574270.404849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574270.468841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574270.980841:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.980849:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574270.980852:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.980861:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6021 00000100:00080000:2.0:1788574270.980871:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.980877:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574270.980911:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864167424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574270.980939:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.980944:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574270.980945:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.980960:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6021 00000100:00080000:2.0:1788574270.980964:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.980969:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574270.980979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574270.980998:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864167552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574270.981000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864167424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574270.981018:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.981023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574270.981025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.981030:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6021 00000100:00080000:2.0:1788574270.981034:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.981038:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574270.981044:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574270.981055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864167680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574270.981064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.981069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788574270.981070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574270.981072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.981079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6021 00000100:00080000:2.0:1788574270.981083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574270.981087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574270.981094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574270.981107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864167808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574270.981118:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff92fc4617c000 x1875449864167424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574270.981119:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574270.981127:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574270.981131:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574270.981138:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6021 00000100:00080000:2.0:1788574270.981142:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574270.981145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864167424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574270.981146:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574270.981154:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574270.981160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574270.981165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574270.981168:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864167936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574270.981172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864167424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574270.981199:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574270.981217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574270.981226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff92fc4617fb80 x1875449864167552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574270.981234:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864167552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574270.981244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574270.981247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574270.981249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574270.981253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574270.981264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574270.981270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff92fc4617c380 x1875449864167680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574270.981277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864167680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574270.981282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574270.981284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574270.981286:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574270.981290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574270.981301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574270.981306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff92fc4617d180 x1875449864167808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574270.981314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864167808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574270.981318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574270.981320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574270.981323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574270.981326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574270.981336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574270.981342:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff92fc4617e680 x1875449864167936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574270.981349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864167936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574270.981354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574270.981356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574270.981357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864167936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574271.428873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574271.492847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574272.452829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574272.516821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574273.476872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574273.540821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574274.500820:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574274.564861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574275.524855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574275.588801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574276.100799:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574276.100807:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574276.100810:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574276.100818:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6026 00000100:00080000:0.0:1788574276.100828:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574276.100834:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574276.100866:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864168064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574276.100903:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574276.100912:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574276.100913:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574276.100918:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6026 00000100:00080000:0.0:1788574276.100926:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574276.100931:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574276.100941:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574276.100942:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864168064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574276.100962:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864168192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574276.100976:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574276.100980:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574276.100982:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574276.100987:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6026 00000100:00080000:0.0:1788574276.101002:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574276.101007:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574276.101011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574276.101023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864168320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574276.101024:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574276.101034:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574276.101037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574276.101039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574276.101042:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6026 00000100:00080000:0.0:1788574276.101047:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574276.101049:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574276.101053:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574276.101063:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864168448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574276.101069:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574276/real 1788574276] req@ffff92fc6704b100 x1875449864168064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574276.101074:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574276.101082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574276.101085:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574276.101089:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6026 00000100:00080000:0.0:1788574276.101092:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574276.101097:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574276.101098:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864168064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574276.101100:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574276.101112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574276.101113:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864168576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574276.101119:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574276.101130:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864168064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574276.101156:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574276.101180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574276.101194:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574276/real 1788574276] req@ffff92fc67049500 x1875449864168192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574276.101206:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864168192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574276.101215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574276.101219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574276.101222:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574276.101229:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574276.101247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574276.101257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574276/real 1788574276] req@ffff92fc67049180 x1875449864168320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574276.101268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864168320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574276.101276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574276.101280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574276.101283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574276.101290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574276.101308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574276.101318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574276/real 1788574276] req@ffff92fc67049c00 x1875449864168448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574276.101330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864168448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574276.101338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574276.101341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574276.101344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574276.101350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574276.101366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574276.101376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574276/real 1788574276] req@ffff92fc67048e00 x1875449864168576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574276.101387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864168576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574276.101394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574276.101398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574276.101401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574276.548827:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574276.612817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574277.572882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574277.636869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574278.596900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574278.660811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574279.620971:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574279.684810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574280.644874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574280.708872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574281.220903:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574281.220914:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574281.220919:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574281.220933:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6031 00000100:00080000:3.0:1788574281.220957:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574281.220969:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574281.221013:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864168704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574281.221040:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574281.221045:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574281.221047:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574281.221053:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6031 00000100:00080000:3.0:1788574281.221059:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574281.221066:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574281.221080:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574281.221101:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864168704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574281.221112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864168832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574281.221127:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574281.221134:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574281.221136:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574281.221140:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6031 00000100:00080000:3.0:1788574281.221146:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574281.221151:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574281.221157:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574281.221168:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864168960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574281.221176:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574281.221179:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574281.221180:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574281.221184:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6031 00000800:00000100:2.0:1788574281.221189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574281.221189:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574281.221197:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574281.221201:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574281.221209:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864169088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574281.221216:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574281.221247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff92fc6733bb80 x1875449864168704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574281.221272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864168704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574281.221282:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574281.221288:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574281.221296:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6031 00000100:00080000:3.0:1788574281.221303:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574281.221312:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788574281.221325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574281.221327:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574281.221335:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574281.221341:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864169216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574281.221344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864168704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574281.221395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574281.221424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574281.221437:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff92fc6733b480 x1875449864168832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574281.221447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864168832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574281.221453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574281.221456:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574281.221459:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574281.221464:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574281.221485:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574281.221493:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff92fc6733aa00 x1875449864168960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574281.221501:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864168960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574281.221508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574281.221511:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574281.221514:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864168960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574281.221519:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574281.221535:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574281.221544:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff92fc67338000 x1875449864169088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574281.221551:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864169088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574281.221556:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574281.221558:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574281.221560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574281.221566:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574281.221584:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574281.221593:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574281/real 1788574281] req@ffff92fc67338700 x1875449864169216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574281.221601:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864169216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574281.221606:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574281.221609:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574281.221612:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574281.668933:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574281.732861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574282.692894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574282.756846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574283.716865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574283.780851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574284.740888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574284.804807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574285.764865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574285.828838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574286.340834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574286.340846:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574286.340850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574286.340861:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6036 00000100:00080000:0.0:1788574286.340871:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574286.340879:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574286.340915:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864169344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574286.340941:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574286.340944:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574286.340946:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574286.340950:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6036 00000100:00080000:0.0:1788574286.340955:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574286.340959:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574286.340970:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574286.340981:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864169344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574286.340989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864169472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574286.341012:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574286.341020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574286.341022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574286.341027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6036 00000100:00080000:0.0:1788574286.341031:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574286.341036:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574286.341041:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788574286.341048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574286.341058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864169600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574286.341069:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574286.341073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574286.341075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574286.341080:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6036 00000100:00080000:0.0:1788574286.341084:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574286.341088:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574286.341095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574286.341095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff92fc6704bb80 x1875449864169344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574286.341112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864169728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574286.341119:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864169344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574286.341127:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574286.341135:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574286.341137:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574286.341138:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574286.341145:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788574286.341148:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6036 00000100:00100000:2.0:1788574286.341154:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864169344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574286.341155:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574286.341164:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574286.341170:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574286.341184:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864169856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574286.341185:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574286.341217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574286.341234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff92fc67048e00 x1875449864169472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574286.341247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864169472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574286.341256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574286.341260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574286.341263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574286.341270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574286.341289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574286.341300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff92fc67049c00 x1875449864169600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574286.341312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864169600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574286.341326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574286.341329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574286.341332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574286.341338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574286.341355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574286.341372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff92fc67049180 x1875449864169728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574286.341385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864169728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574286.341393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574286.341396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574286.341399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574286.341416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574286.341434:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574286.341444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574286/real 1788574286] req@ffff92fc67049500 x1875449864169856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574286.341455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864169856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574286.341463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574286.341466:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574286.341470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864169856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574286.788851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574286.852786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574287.812845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574287.876859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574288.836869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574288.900846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574289.860876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574289.924892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574290.884878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574290.948840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574291.460829:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.460840:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574291.460846:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.460857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6042 00000100:00080000:0.0:1788574291.460867:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.460874:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574291.460907:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864169984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574291.460932:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.460935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574291.460937:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.460939:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6042 00000100:00080000:0.0:1788574291.460942:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.460945:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574291.460953:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574291.460962:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864170112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574291.460970:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.460975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574291.460975:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864169984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574291.460979:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.460986:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6042 00000100:00080000:0.0:1788574291.460988:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.460990:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574291.460993:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574291.461003:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864170240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574291.461010:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.461012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574291.461014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.461016:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6042 00000100:00080000:0.0:1788574291.461019:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.461021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574291.461024:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574291.461034:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864170368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574291.461038:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574291.461046:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574291.461049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574291.461051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574291.461055:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6042 00000100:00080000:0.0:1788574291.461057:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574291.461060:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574291.461063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574291.461072:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864170496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574291.461078:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff92fc6704b100 x1875449864169984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574291.461092:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864169984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574291.461101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574291.461104:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574291.461107:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864169984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574291.461126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574291.461142:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574291.461150:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff92fc67049880 x1875449864170112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574291.461163:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864170112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574291.461169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574291.461172:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574291.461174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574291.461178:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574291.461189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574291.461197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff92fc6704ad80 x1875449864170240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574291.461204:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864170240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574291.461212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574291.461215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574291.461219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574291.461225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574291.461245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574291.461257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff92fc6704a680 x1875449864170368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574291.461270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864170368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574291.461276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574291.461278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574291.461280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574291.461284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574291.461298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574291.461305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574291/real 1788574291] req@ffff92fc6704aa00 x1875449864170496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574291.461312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864170496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574291.461318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574291.461320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574291.461322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574291.908823:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574291.972848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574292.932887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574292.996828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574293.956799:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574294.020805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574294.980938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574295.044904:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574296.004936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574296.068846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574296.580828:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574296.580840:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574296.580844:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574296.580852:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6047 00000100:00080000:0.0:1788574296.580861:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574296.580869:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574296.580900:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864170624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574296.580922:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574296.580925:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574296.580926:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574296.580929:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6047 00000100:00080000:0.0:1788574296.580931:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574296.580933:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574296.580940:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574296.580955:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864170752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574296.580963:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574296.580967:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574296.580969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864170624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574296.580970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574296.580979:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6047 00000100:00080000:0.0:1788574296.580981:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574296.580984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574296.580988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574296.580997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864170880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574296.581003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574296.581006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574296.581008:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574296.581012:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6047 00000100:00080000:0.0:1788574296.581017:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574296.581021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574296.581026:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574296.581035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574296.581042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864171008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574296.581054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574296.581058:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574296.581060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574296.581064:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6047 00000100:00080000:0.0:1788574296.581068:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574296.581071:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574296.581076:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574296.581079:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff92fc6704a300 x1875449864170624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574296.581089:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864171136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574296.581099:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864170624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574296.581108:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574296.581112:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574296.581115:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864170624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574296.581135:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574296.581152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574296.581161:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff92fc67048000 x1875449864170752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574296.581170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864170752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574296.581176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574296.581179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574296.581183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574296.581191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574296.581220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574296.581232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff92fc67049f80 x1875449864170880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574296.581243:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864170880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574296.581248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574296.581250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574296.581253:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864170880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574296.581257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574296.581271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574296.581278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff92fc67048700 x1875449864171008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574296.581286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864171008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574296.581291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574296.581293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574296.581295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574296.581299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574296.581309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574296.581315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff92fc67048380 x1875449864171136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574296.581322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864171136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574296.581327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574296.581329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574296.581331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574297.028888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574297.092820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574298.052879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574298.116806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574299.076864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574299.140831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574300.100870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574300.164790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574301.124940:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574301.188867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574301.700915:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574301.700930:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574301.700936:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574301.700950:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6052 00000100:00080000:2.0:1788574301.700964:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574301.700973:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574301.701025:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864171264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574301.701067:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574301.701072:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574301.701074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574301.701078:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6052 00000100:00080000:2.0:1788574301.701082:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574301.701088:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574301.701101:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574301.701117:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864171392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574301.701120:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864171264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574301.701143:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574301.701147:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574301.701149:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574301.701154:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6052 00000100:00080000:2.0:1788574301.701158:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574301.701163:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574301.701169:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574301.701182:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864171520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574301.701192:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574301.701195:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788574301.701202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574301.701208:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574301.701215:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6052 00000100:00080000:2.0:1788574301.701220:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574301.701225:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574301.701230:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574301.701241:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864171648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574301.701250:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574301.701253:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574301.701255:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574301.701259:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6052 00000100:00080000:2.0:1788574301.701263:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574301.701267:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574301.701272:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574301.701283:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864171776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574301.701340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff92fc4617dc00 x1875449864171264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.701365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864171264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.701374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.701378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574301.701381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864171264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574301.701408:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574301.701429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574301.701443:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff92fc4617ca80 x1875449864171392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.701456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864171392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.701466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.701470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574301.701475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574301.701481:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574301.701500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574301.701508:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff92fc4617c700 x1875449864171520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.701516:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864171520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.701522:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.701524:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574301.701526:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574301.701530:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574301.701541:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574301.701547:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff92fc4617f800 x1875449864171648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.701555:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864171648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.701561:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.701563:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574301.701565:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574301.701568:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574301.701578:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574301.701584:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff92fc4617f100 x1875449864171776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574301.701606:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864171776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574301.701611:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574301.701613:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574301.701616:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864171776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574302.148908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574302.212806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574303.172891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574303.236832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574304.196916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574304.260821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574305.220890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574305.284808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574306.244877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574306.308848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574306.820849:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574306.820864:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574306.820870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574306.820884:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6057 00000100:00080000:2.0:1788574306.820899:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574306.820910:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574306.820965:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864171904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574306.821001:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574306.821006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574306.821009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574306.821013:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6057 00000100:00080000:2.0:1788574306.821018:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574306.821023:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574306.821036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574306.821054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864172032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574306.821060:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864171904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574306.821075:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574306.821083:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574306.821085:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574306.821090:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6057 00000100:00080000:2.0:1788574306.821096:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574306.821101:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574306.821108:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574306.821124:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864172160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574306.821136:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574306.821140:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574306.821142:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574306.821147:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6057 00000800:00000100:1.0:1788574306.821154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574306.821155:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574306.821163:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574306.821170:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574306.821185:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864172288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574306.821196:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574306.821200:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574306.821203:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574306.821207:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6057 00000100:00100000:1.0:1788574306.821212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff92fc4617f480 x1875449864171904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574306.821214:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574306.821224:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574306.821233:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574306.821238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864171904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574306.821250:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864172416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574306.821255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.821263:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574306.821272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864171904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574306.821304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574306.821332:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574306.821347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff92fc4617ed80 x1875449864172032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.821361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864172032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.821370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.821373:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574306.821376:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574306.821381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574306.821398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574306.821407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff92fc4617f100 x1875449864172160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.821420:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864172160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.821428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.821432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574306.821435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574306.821442:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574306.821465:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574306.821477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff92fc4617f800 x1875449864172288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.821490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864172288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.821500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.821503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574306.821507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574306.821512:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574306.821537:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574306.821546:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff92fc4617c700 x1875449864172416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574306.821554:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864172416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574306.821562:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574306.821566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574306.821569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574307.268857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574307.332839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574308.292857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574308.356845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574309.316816:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574309.380811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574310.340924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574310.404868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574311.364867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574311.428860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574311.940845:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574311.940857:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574311.940862:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574311.940873:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6062 00000100:00080000:2.0:1788574311.940885:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574311.940893:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574311.940938:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864172544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574311.940965:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574311.940969:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574311.940970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574311.940973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6062 00000100:00080000:2.0:1788574311.940976:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574311.940978:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574311.940987:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574311.940997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864172672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574311.941005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574311.941008:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574311.941009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574311.941012:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6062 00000100:00080000:2.0:1788574311.941016:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574311.941019:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574311.941025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864172544:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574311.941025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574311.941040:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864172800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574311.941047:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574311.941050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574311.941052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574311.941055:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6062 00000100:00080000:2.0:1788574311.941057:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574311.941060:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574311.941064:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574311.941072:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864172928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574311.941078:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574311.941080:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574311.941081:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574311.941084:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6062 00000100:00080000:2.0:1788574311.941086:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574311.941090:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574311.941092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574311.941099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864173056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574311.941106:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574311.941156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff92fc4617ca80 x1875449864172544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.941173:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864172544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.941183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.941186:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574311.941189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864172544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574311.941195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574311.941211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574311.941219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff92fc4617dc00 x1875449864172672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.941228:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864172672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.941234:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.941236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574311.941238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574311.941262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574311.941279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574311.941290:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff92fc4617e680 x1875449864172800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.941303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864172800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.941319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.941322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574311.941326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574311.941331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574311.941348:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574311.941356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff92fc4617d180 x1875449864172928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.941364:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864172928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.941369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.941371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574311.941374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864172928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574311.941377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574311.941387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574311.941393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff92fc4617c380 x1875449864173056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574311.941400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864173056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574311.941405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574311.941407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574311.941410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574312.388846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574312.452830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574313.412825:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574313.476854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574314.436955:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574314.500898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574315.460866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574315.524828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574316.484907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574316.548808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574317.060869:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574317.060875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574317.060879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574317.060889:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6067 00000100:00080000:2.0:1788574317.060899:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574317.060906:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574317.060936:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864173184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574317.060960:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574317.060963:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574317.060965:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574317.060968:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6067 00000100:00080000:2.0:1788574317.060972:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574317.060974:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574317.060983:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574317.060996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864173312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574317.061002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864173184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574317.061009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574317.061013:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574317.061014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574317.061018:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6067 00000100:00080000:2.0:1788574317.061020:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574317.061022:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574317.061027:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574317.061040:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864173440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574317.061047:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574317.061050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574317.061051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574317.061055:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6067 00000100:00080000:2.0:1788574317.061059:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574317.061063:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574317.061068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788574317.061072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574317.061084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864173568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574317.061091:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574317.061094:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574317.061096:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574317.061099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6067 00000100:00080000:2.0:1788574317.061102:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574317.061105:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574317.061109:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574317.061110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574317/real 1788574317] req@ffff92fc4617fb80 x1875449864173184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574317.061119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864173696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574317.061135:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864173184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574317.061209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574317.061215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574317.061220:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864173184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574317.061244:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574317.061270:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574317.061284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574317/real 1788574317] req@ffff92fc4617c000 x1875449864173312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574317.061295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864173312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574317.061301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574317.061303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574317.061306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574317.061310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574317.061324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574317.061332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574317/real 1788574317] req@ffff92fc4617d880 x1875449864173440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574317.061339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864173440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574317.061458:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574317.061461:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574317.061463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574317.061468:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574317.061497:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574317.061509:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574317/real 1788574317] req@ffff92fc4617d500 x1875449864173568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574317.061582:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864173568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574317.061588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574317.061591:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574317.061593:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574317.061598:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574317.061613:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574317.061621:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574317/real 1788574317] req@ffff92fc4617e300 x1875449864173696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574317.061633:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864173696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574317.061638:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574317.061640:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574317.061642:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574317.508875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574317.572826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574318.532861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574318.596843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574319.556881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574319.620826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574320.580879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574320.644853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574321.604895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574321.668847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574322.180916:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.180934:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574322.180939:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.180949:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6072 00000100:00080000:3.0:1788574322.180961:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.180969:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574322.181010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864173824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574322.181038:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.181041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574322.181043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.181046:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6072 00000100:00080000:3.0:1788574322.181048:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.181050:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574322.181059:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574322.181070:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864173952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574322.181083:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.181086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574322.181087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574322.181093:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864173824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574322.181094:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6072 00000100:00080000:3.0:1788574322.181100:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.181104:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574322.181107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574322.181115:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864174080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574322.181122:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.181125:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574322.181126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.181128:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6072 00000100:00080000:3.0:1788574322.181131:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.181133:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574322.181136:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574322.181146:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864174208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574322.181152:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574322.181155:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574322.181156:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574322.181159:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6072 00000100:00080000:3.0:1788574322.181163:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574322.181165:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574322.181169:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574322.181173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574322.181179:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864174336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574322.181228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff92fc6733a680 x1875449864173824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.181250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864173824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.181264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.181269:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574322.181273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864173824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574322.181308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574322.181329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574322.181338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff92fc67338e00 x1875449864173952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.181347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864173952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.181354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.181358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574322.181373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864173952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574322.181385:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574322.181409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574322.181423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff92fc6733a300 x1875449864174080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.181436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864174080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.181445:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.181449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574322.181453:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574322.181459:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574322.181484:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574322.181497:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff92fc67338700 x1875449864174208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.181511:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864174208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.181520:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.181523:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574322.181527:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574322.181534:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574322.181556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574322.181568:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574322/real 1788574322] req@ffff92fc67338000 x1875449864174336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574322.181580:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864174336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574322.181589:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574322.181593:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574322.181596:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574322.628985:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574322.692910:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574323.652961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574323.716819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574324.676925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574324.740877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574325.700920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574325.764909:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574326.724907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574326.788862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574327.300885:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574327.300895:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574327.300901:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574327.300911:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6077 00000100:00080000:2.0:1788574327.300923:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574327.300930:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574327.300968:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864174464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574327.300995:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574327.300999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574327.301001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574327.301005:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6077 00000100:00080000:2.0:1788574327.301009:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574327.301013:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574327.301023:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574327.301034:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864174592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574327.301044:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574327.301048:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574327.301051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574327.301055:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6077 00000100:00080000:2.0:1788574327.301059:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574327.301065:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574327.301066:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864174464:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574327.301071:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574327.301086:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864174720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574327.301103:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574327.301106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574327.301108:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574327.301113:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6077 00000100:00080000:2.0:1788574327.301117:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574327.301121:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574327.301126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574327.301139:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864174848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574327.301148:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574327.301153:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174592:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788574327.301154:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574327.301160:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574327.301166:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6077 00000100:00080000:2.0:1788574327.301172:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574327.301177:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574327.301182:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574327.301188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574327.301197:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864174976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574327.301208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574327.301228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574327.748891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574327.812882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574328.772882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574328.836873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574329.796865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574329.860825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788574330.373076:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788574330.373128:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574330.373146:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574330.373194:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574330.373205:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574330.373215:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574330.373226:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788574330.373265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff92fc4617d880 x1875449864174976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.373300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864174976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.373318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.373326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574330.373332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174976:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574330.373343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff92fc4617e300 x1875449864174720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.373360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864174720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.373378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.373383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574330.373405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574330.373414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff92fc4617d500 x1875449864174848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.373432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864174848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.373442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.373447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574330.373451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574330.373459:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff92fc4617ea00 x1875449864174464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.373474:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864174464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.373483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.373487:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574330.373491:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864174464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574330.373500:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574327/real 1788574330] req@ffff92fc4617ce00 x1875449864174592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574330.373514:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864174592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574330.373525:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574330.373529:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574330.373534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864174592:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788574330.820922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574330.884875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574331.844951:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574331.908911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574332.420793:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.420805:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574332.420809:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.420820:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6083 00000100:00080000:2.0:1788574332.420832:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.420841:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574332.420893:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864175104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574332.420924:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.420929:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574332.420931:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.420934:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6083 00000100:00080000:2.0:1788574332.420938:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.420943:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574332.420955:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574332.420970:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864175232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574332.420982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.420987:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788574332.420990:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864175104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574332.420991:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.421001:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6083 00000100:00080000:2.0:1788574332.421005:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.421009:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574332.421015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574332.421028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864175360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574332.421038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.421042:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574332.421044:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.421048:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6083 00000100:00080000:2.0:1788574332.421053:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.421057:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574332.421063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574332.421075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864175488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574332.421086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574332.421090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574332.421092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574332.421099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6083 00000800:00000100:1.0:1788574332.421100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574332.421106:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574332.421110:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574332.421115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574332.421126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864175616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574332.421161:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff92fc4617c000 x1875449864175104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.421185:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864175104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.421197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.421201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574332.421205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864175104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574332.421242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574332.421267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574332.421292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff92fc4617fb80 x1875449864175232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.421306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864175232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.421314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.421320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574332.421323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574332.421330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574332.421350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574332.421362:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff92fc4617c380 x1875449864175360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.421375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864175360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.421383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.421389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574332.421393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574332.421399:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574332.421420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574332.421434:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff92fc4617d180 x1875449864175488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.421447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864175488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.421456:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.421460:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574332.421463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574332.421471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574332.421493:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574332.421504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574332/real 1788574332] req@ffff92fc4617e680 x1875449864175616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574332.421517:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864175616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574332.421528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574332.421531:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574332.421534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574332.868917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574332.932860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574333.892947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574333.956922:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574334.916915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574334.980876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574335.941004:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574336.004902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574336.964861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574337.028817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574337.540881:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.540896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574337.540902:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.540915:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6088 00000100:00080000:2.0:1788574337.540953:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.540963:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574337.541017:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864175744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.541052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.541056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574337.541058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.541061:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6088 00000100:00080000:2.0:1788574337.541064:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.541068:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574337.541079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574337.541091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864175872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.541100:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.541104:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574337.541105:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.541109:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6088 00000100:00080000:2.0:1788574337.541112:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.541114:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574337.541116:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574337.541123:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864176000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.541132:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.541135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574337.541137:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.541143:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6088 00000100:00100000:1.0:1788574337.541145:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864175744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574337.541147:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.541157:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574337.541166:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574337.541178:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864176128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574337.541186:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574337.541189:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574337.541190:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574337.541194:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6088 00000100:00080000:2.0:1788574337.541196:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574337.541206:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574337.541215:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574337.541223:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864176256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574337.541236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574337.541291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff92fc4617dc00 x1875449864175744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.541307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864175744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.541317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.541320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574337.541323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864175744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574337.541331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574337.541349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574337.541358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff92fc4617ca80 x1875449864175872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.541366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864175872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.541377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.541380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574337.541382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864175872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574337.541404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574337.541416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574337.541423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff92fc4617c700 x1875449864176000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.541431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864176000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.541436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.541438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574337.541441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574337.541444:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574337.541455:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574337.541461:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff92fc4617f800 x1875449864176128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.541468:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864176128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.541474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.541477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574337.541481:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574337.541488:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574337.541508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574337.541521:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574337/real 1788574337] req@ffff92fc4617f100 x1875449864176256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574337.541534:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864176256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574337.541543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574337.541546:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574337.541550:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574337.988870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574338.052845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574339.012911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574339.076790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574340.036901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574340.100840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574341.060895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574341.124777:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574342.084941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574342.148786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574342.660900:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574342.660913:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574342.660920:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574342.660941:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6093 00000100:00080000:3.0:1788574342.660955:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574342.660965:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574342.661023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864176384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574342.661050:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574342.661053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574342.661055:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574342.661058:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6093 00000100:00080000:3.0:1788574342.661061:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574342.661064:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574342.661074:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574342.661084:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864176512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574342.661093:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574342.661098:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574342.661099:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574342.661102:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6093 00000100:00080000:3.0:1788574342.661105:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574342.661107:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574342.661111:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574342.661113:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864176384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574342.661121:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864176640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574342.661132:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574342.661135:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574342.661138:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574342.661142:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6093 00000100:00080000:3.0:1788574342.661146:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574342.661151:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574342.661157:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574342.661170:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864176768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574342.661181:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574342.661184:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574342.661185:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574342.661190:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6093 00000100:00080000:3.0:1788574342.661193:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574342.661196:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574342.661203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574342.661204:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574342.661220:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864176896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574342.661261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff92fc6733aa00 x1875449864176384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.661278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864176384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.661287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.661290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574342.661294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864176384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574342.661299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574342.661319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574342.661329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff92fc6733b480 x1875449864176512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.661338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864176512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.661344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.661346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574342.661349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574342.661373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574342.661385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574342.661392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff92fc6733bb80 x1875449864176640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.661399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864176640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.661405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.661407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574342.661410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574342.661413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574342.661424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574342.661430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff92fc6733b100 x1875449864176768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.661437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864176768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.661443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.661445:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574342.661448:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574342.661451:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574342.661462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574342.661467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574342] req@ffff92fc67339500 x1875449864176896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574342.661474:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864176896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574342.661480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574342.661482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574342.661484:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864176896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574343.108899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574343.172797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574344.132844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574344.196841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574345.156924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574345.220864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574346.180949:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574346.244882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574347.204920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574347.268861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574347.780807:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.780815:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574347.780819:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.780827:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6098 00000100:00080000:3.0:1788574347.780836:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.780842:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574347.780881:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864177024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.780902:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.780904:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574347.780906:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.780908:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6098 00000100:00080000:3.0:1788574347.780910:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.780913:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574347.780920:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574347.780930:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864177152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.780937:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.780940:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574347.780941:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.780943:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6098 00000100:00080000:3.0:1788574347.780946:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.780949:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574347.780952:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574347.780957:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864177024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574347.780962:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864177280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.780971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.780973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574347.780974:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.780977:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6098 00000100:00080000:3.0:1788574347.780980:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.780981:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574347.780985:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574347.780991:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864177408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574347.780998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574347.781000:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574347.781002:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574347.781004:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6098 00000100:00080000:3.0:1788574347.781016:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574347.781037:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574347.781043:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574347.781047:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574347.781063:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864177536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574347.781092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff92fc67338380 x1875449864177024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.781114:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864177024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.781126:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.781131:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574347.781135:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864177024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574347.781144:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574347.781167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574347.781179:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff92fc67338a80 x1875449864177152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.781192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864177152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.781203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.781207:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574347.781210:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574347.781234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574347.781253:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574347.781265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff92fc6733b800 x1875449864177280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.781278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864177280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.781287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.781291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574347.781294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574347.781300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574347.781319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574347.781330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff92fc67339880 x1875449864177408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.781343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864177408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.781352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.781356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574347.781360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574347.781366:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574347.781384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574347.781395:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff92fc67339c00 x1875449864177536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.781407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864177536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.781416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574347.781420:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574347.781423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574348.228827:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574348.292838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574349.252884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574349.316824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574350.276870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574350.340865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574351.300855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574351.364789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574352.324849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574352.388821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574352.900821:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574352.900830:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574352.900834:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574352.900844:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6103 00000100:00080000:3.0:1788574352.900852:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574352.900860:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574352.900892:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864177664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574352.900921:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574352.900926:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574352.900928:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574352.900934:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6103 00000100:00080000:3.0:1788574352.900939:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574352.900945:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574352.900955:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574352.900959:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864177664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574352.900973:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864177792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574352.900987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574352.900990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574352.900992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574352.900995:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6103 00000100:00080000:3.0:1788574352.900998:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574352.901002:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574352.901016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574352.901030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574352.901030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864177920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574352.901040:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574352.901043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574352.901045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574352.901049:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6103 00000100:00080000:3.0:1788574352.901053:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574352.901057:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574352.901061:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574352.901070:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864178048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574352.901090:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574352.901093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574352.901094:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574352.901098:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6103 00000100:00080000:3.0:1788574352.901101:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574352.901105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff92fc6733ad80 x1875449864177664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574352.901106:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574352.901112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574352.901123:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864178176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574352.901149:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864177664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574352.901162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574352.901174:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574352.901178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864177664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574352.901202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574352.901228:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574352.901238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff92fc67339f80 x1875449864177792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574352.901246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864177792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574352.901253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574352.901255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574352.901257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574352.901262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574352.901282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574352.901293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff92fc67339180 x1875449864177920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574352.901305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864177920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574352.901314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574352.901318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574352.901321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864177920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574352.901328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574352.901349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574352.901362:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff92fc67339c00 x1875449864178048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574352.901376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864178048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574352.901393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574352.901396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574352.901399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574352.901404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574352.901426:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574352.901442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff92fc67339880 x1875449864178176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574352.901453:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864178176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574352.901461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574352.901464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574352.901468:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574353.348889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574353.412841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574354.372871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574354.436841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574355.396824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574355.460822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574356.420936:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574356.484890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574357.444905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574357.508879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574358.020830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574358.020838:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574358.020843:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574358.020852:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6108 00000100:00080000:3.0:1788574358.020860:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574358.020867:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574358.020900:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864178304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574358.020922:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574358.020926:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574358.020927:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574358.020929:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6108 00000100:00080000:3.0:1788574358.020932:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574358.020936:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574358.020946:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574358.020960:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864178432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574358.020972:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574358.020976:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574358.020978:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574358.020983:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6108 00000100:00100000:0.0:1788574358.020985:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864178304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574358.020987:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574358.020994:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574358.021000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574358.021019:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864178560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574358.021034:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574358.021037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574358.021038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574358.021041:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6108 00000100:00080000:3.0:1788574358.021044:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574358.021046:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574358.021050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574358.021057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864178688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574358.021063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574358.021065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574358.021067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574358.021072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574358.021072:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6108 00000100:00080000:3.0:1788574358.021075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574358.021078:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574358.021082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574358.021089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864178816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574358.021116:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574358/real 1788574358] req@ffff92fc6733b800 x1875449864178304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574358.021154:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864178304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574358.021162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574358.021165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574358.021168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864178304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574358.021174:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574358.021192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574358.021201:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574358/real 1788574358] req@ffff92fc67338a80 x1875449864178432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574358.021209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864178432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574358.021215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574358.021217:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574358.021219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574358.021237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574358.021249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574358.021264:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574358/real 1788574358] req@ffff92fc67338380 x1875449864178560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574358.021272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864178560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574358.021282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574358.021285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574358.021287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574358.021290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574358.021301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574358.021308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574358/real 1788574358] req@ffff92fc67339500 x1875449864178688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574358.021315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864178688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574358.021320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574358.021322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574358.021324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574358.021328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574358.021338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574358.021344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574358/real 1788574358] req@ffff92fc6733b100 x1875449864178816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574358.021351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864178816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574358.021356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574358.021358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574358.021360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864178816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574358.468857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574358.532798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574359.492849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574359.556835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574360.516856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574360.580805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574361.540845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574361.604810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574362.564897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574362.628812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574363.140862:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574363.140872:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574363.140877:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574363.140887:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6113 00000100:00080000:2.0:1788574363.140899:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574363.140907:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574363.140948:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864178944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574363.140979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574363.140987:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574363.140989:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574363.140994:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6113 00000100:00080000:2.0:1788574363.140999:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574363.141005:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574363.141018:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574363.141033:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864178944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574363.141037:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864179072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574363.141057:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574363.141069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574363.141072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574363.141078:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6113 00000100:00080000:2.0:1788574363.141084:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574363.141088:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574363.141094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574363.141101:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574363.141113:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864179200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574363.141125:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574363.141129:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574363.141131:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574363.141137:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6113 00000100:00080000:2.0:1788574363.141142:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574363.141143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574363/real 1788574363] req@ffff92fc4617ed80 x1875449864178944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574363.141151:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574363.141163:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574363.141165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864178944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574363.141195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788574363.141200:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864179328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574363.141201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574363.141211:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864178944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574363.141227:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574363.141231:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574363.141235:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574363.141236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179072:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788574363.141249:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6113 00000100:00080000:2.0:1788574363.141254:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574363.141260:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788574363.141266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788574363.141268:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574363.141279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574363/real 1788574363] req@ffff92fc4617f480 x1875449864179072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574363.141289:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864179456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574363.141290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864179072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574363.141307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574363.141312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574363.141326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574363.141347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574363.141374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574363.141388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574363/real 1788574363] req@ffff92fc4617f100 x1875449864179200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574363.141402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864179200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574363.141411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574363.141415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574363.141417:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574363.141423:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574363.141440:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574363.141447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574363/real 1788574363] req@ffff92fc4617f800 x1875449864179328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574363.141455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864179328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574363.141460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574363.141462:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574363.141465:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574363.141478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574363.141489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574363.141496:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574363/real 1788574363] req@ffff92fc4617c700 x1875449864179456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574363.141503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864179456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574363.141508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574363.141510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574363.141512:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574363.588869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574363.652823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574364.612912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574364.676897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574365.636852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574365.700828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574366.660898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574366.724829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574367.684819:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574367.748833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574368.260887:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574368.260901:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574368.260907:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574368.260919:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6118 00000100:00080000:3.0:1788574368.260935:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574368.260944:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574368.260994:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864179584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574368.261033:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574368.261037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574368.261040:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574368.261044:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6118 00000100:00080000:3.0:1788574368.261048:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574368.261052:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574368.261064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574368.261080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864179712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574368.261089:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864179584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574368.261096:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574368.261101:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574368.261103:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574368.261108:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6118 00000100:00080000:3.0:1788574368.261113:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574368.261117:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574368.261123:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574368.261137:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864179840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574368.261148:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574368.261153:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574368.261155:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574368.261159:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6118 00000100:00080000:3.0:1788574368.261163:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574368.261167:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788574368.261172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574368.261173:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574368.261188:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864179968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574368.261199:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574368.261203:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574368.261205:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574368.261209:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6118 00000100:00080000:3.0:1788574368.261213:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574368.261217:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574368.261238:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574368.261251:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864180096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574368.261259:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff92fc6733bb80 x1875449864179584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574368.261303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864179584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574368.261316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574368.261321:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574368.261326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864179584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574368.261366:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574368.261393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574368.261408:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff92fc6733b480 x1875449864179712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574368.261422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864179712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574368.261432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574368.261436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574368.261440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574368.261447:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574368.261470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574368.261482:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff92fc6733aa00 x1875449864179840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574368.261495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864179840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574368.261511:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574368.261515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574368.261519:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574368.261526:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574368.261547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574368.261559:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff92fc67338000 x1875449864179968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574368.261572:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864179968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574368.261580:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574368.261584:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574368.261588:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864179968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574368.261595:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574368.261615:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574368.261627:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574368/real 1788574368] req@ffff92fc67338700 x1875449864180096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574368.261640:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864180096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574368.261648:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574368.261652:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574368.261656:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574368.708935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574368.772880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574369.732887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574369.796895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574370.756908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574370.820846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574371.780810:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574371.844827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574372.804864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574372.868837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574373.380824:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.380832:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574373.380835:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.380843:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6123 00000100:00080000:0.0:1788574373.380852:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.380858:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574373.380889:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864180224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.380920:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.380923:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574373.380924:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.380927:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6123 00000100:00080000:0.0:1788574373.380929:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.380931:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574373.380939:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574373.380951:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864180352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.380959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.380963:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788574373.380964:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864180224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574373.380965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.380971:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6123 00000100:00080000:0.0:1788574373.380976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.380978:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574373.380981:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574373.380990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864180480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.380996:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.380999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574373.381000:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.381003:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6123 00000100:00080000:0.0:1788574373.381017:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.381020:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574373.381023:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574373.381027:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574373.381035:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864180608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574373.381043:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574373.381045:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574373.381047:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574373.381050:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6123 00000100:00080000:0.0:1788574373.381052:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574373.381055:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574373.381059:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574373.381068:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff92fc67048a80 x1875449864180224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574373.381069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864180736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574373.381104:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864180224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574373.381120:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574373.381124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574373.381129:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864180224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574373.381152:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574373.381176:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574373.381186:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff92fc67048380 x1875449864180352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574373.381195:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864180352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574373.381201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574373.381203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574373.381205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574373.381210:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574373.381221:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574373.381228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff92fc67048700 x1875449864180480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574373.381235:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864180480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574373.381240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574373.381242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574373.381244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574373.381249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574373.381259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574373.381265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff92fc67049f80 x1875449864180608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574373.381273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864180608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574373.381291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574373.381294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574373.381296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574373.381300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574373.381314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574373.381321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574373/real 1788574373] req@ffff92fc67048000 x1875449864180736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574373.381329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864180736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574373.381334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574373.381337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574373.381338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574373.828881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574373.892815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574374.852866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574374.916836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574375.876867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574375.940836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574376.900849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574376.964842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574377.924862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574377.988799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574378.500843:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574378.500850:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574378.500854:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574378.500862:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6129 00000100:00080000:3.0:1788574378.500871:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574378.500877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574378.500918:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864180864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574378.500939:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574378.500942:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574378.500944:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574378.500946:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6129 00000100:00080000:3.0:1788574378.500952:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574378.500954:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574378.500962:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574378.500971:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864180992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574378.500979:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574378.500981:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574378.500983:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574378.500985:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6129 00000100:00080000:3.0:1788574378.500987:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574378.500991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864180864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574378.500992:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574378.500999:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574378.501006:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864181120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574378.501013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574378.501016:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574378.501017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574378.501019:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6129 00000100:00080000:3.0:1788574378.501022:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574378.501025:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574378.501028:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574378.501037:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864181248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574378.501048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574378.501051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574378.501052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574378.501054:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6129 00000100:00080000:3.0:1788574378.501057:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574378.501062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574378.501063:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574378.501070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574378.501077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864181376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574378.501108:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff92fc6733a300 x1875449864180864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574378.501129:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864180864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574378.501146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574378.501151:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574378.501155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864180864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574378.501163:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574378.501188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574378.501201:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff92fc67338e00 x1875449864180992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574378.501213:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864180992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574378.501223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574378.501226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574378.501229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864180992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574378.501255:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574378.501273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574378.501282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff92fc6733a680 x1875449864181120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574378.501290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864181120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574378.501300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574378.501302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574378.501304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574378.501308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574378.501319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574378.501325:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff92fc67338700 x1875449864181248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574378.501355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864181248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574378.501363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574378.501367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574378.501370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574378.501377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574378.501398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574378.501407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574378/real 1788574378] req@ffff92fc67338000 x1875449864181376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574378.501415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864181376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574378.501420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574378.501422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574378.501425:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574378.948835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574379.012842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574379.972893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574380.036779:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574380.996903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574381.060830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574382.020942:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574382.084895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574383.044941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574383.108837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574383.620857:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574383.620865:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574383.620869:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574383.620877:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6134 00000100:00080000:2.0:1788574383.620886:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574383.620892:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574383.620928:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864181504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574383.620951:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574383.620954:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574383.620956:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574383.620958:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6134 00000100:00080000:2.0:1788574383.620961:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574383.620963:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574383.620971:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574383.620980:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864181632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574383.620987:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574383.620989:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574383.620991:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574383.620993:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6134 00000100:00080000:2.0:1788574383.620995:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574383.621000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864181504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574383.621000:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574383.621006:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574383.621014:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864181760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574383.621020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574383.621022:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574383.621024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574383.621026:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6134 00000100:00080000:2.0:1788574383.621028:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574383.621030:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574383.621035:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574383.621046:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864181888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574383.621063:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574383.621067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574383.621069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574383.621073:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6134 00000800:00000100:1.0:1788574383.621077:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574383.621080:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574383.621087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574383.621093:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574383.621107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864182016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574383.621124:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff92fc4617ca80 x1875449864181504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574383.621147:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864181504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574383.621158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574383.621161:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574383.621164:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864181504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574383.621170:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574383.621189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574383.621199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff92fc4617dc00 x1875449864181632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574383.621207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864181632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574383.621213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574383.621216:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574383.621218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574383.621237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574383.621249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574383.621256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff92fc4617e680 x1875449864181760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574383.621265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864181760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574383.621270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574383.621272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574383.621274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574383.621278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574383.621288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574383.621294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff92fc4617d180 x1875449864181888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574383.621302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864181888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574383.621307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574383.621309:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574383.621311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864181888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574383.621314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574383.621324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574383.621330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff92fc4617c380 x1875449864182016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574383.621337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864182016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574383.621342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574383.621344:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574383.621346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574384.068953:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574384.132801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574385.092872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574385.156837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574386.116858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574386.180825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574387.140811:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574387.204840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574388.164858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574388.228838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574388.740822:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574388.740833:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574388.740837:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574388.740846:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6139 00000100:00080000:2.0:1788574388.740856:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574388.740862:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574388.740897:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864182144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574388.740919:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574388.740922:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574388.740924:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574388.740932:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6139 00000100:00080000:2.0:1788574388.740935:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574388.740939:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574388.740951:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574388.740967:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864182144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574388.740972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864182272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574388.740987:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574388.740991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574388.740994:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574388.740999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6139 00000100:00080000:2.0:1788574388.741006:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574388.741021:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574388.741028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574388.741040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574388.741045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864182400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574388.741059:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574388.741063:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574388.741066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574388.741070:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6139 00000100:00080000:2.0:1788574388.741075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574388.741080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574388.741087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574388.741089:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff92fc4617fb80 x1875449864182144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574388.741104:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864182528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574388.741115:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864182144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574388.741118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574388.741128:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574388.741133:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574388.741141:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6139 00000100:00080000:0.0:1788574388.741143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574388.741145:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574388.741153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574388.741158:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574388.741166:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574388.741169:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864182656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574388.741176:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864182144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574388.741209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574388.741234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574388.741244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff92fc4617c000 x1875449864182272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574388.741255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864182272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574388.741262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574388.741265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574388.741276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574388.741284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574388.741308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574388.741325:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff92fc4617ce00 x1875449864182400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574388.741353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864182400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574388.741363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574388.741367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574388.741380:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574388.741388:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574388.741413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574388.741424:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff92fc4617ea00 x1875449864182528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574388.741432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864182528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574388.741479:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574388.741483:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574388.741487:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574388.741493:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574388.741510:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574388.741519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff92fc4617d500 x1875449864182656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574388.741530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864182656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574388.741545:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574388.741549:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574388.741552:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574389.188790:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574389.252870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574390.212861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574390.276827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574391.236854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574391.300810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574392.260843:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574392.324816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574393.284803:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574393.348785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574393.860834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574393.860843:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574393.860847:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574393.860855:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6144 00000100:00080000:3.0:1788574393.860865:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574393.860871:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574393.860905:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864182784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574393.860929:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574393.860933:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574393.860935:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574393.860939:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6144 00000100:00080000:3.0:1788574393.860942:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574393.860946:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574393.860956:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574393.860969:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864182912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574393.860980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574393.860983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574393.860984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574393.860988:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6144 00000100:00080000:3.0:1788574393.860991:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574393.860993:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864182784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574393.860995:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574393.861000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574393.861009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864183040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574393.861019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574393.861022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574393.861024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574393.861027:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6144 00000100:00080000:3.0:1788574393.861030:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574393.861033:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574393.861038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574393.861046:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864183168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574393.861055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574393.861058:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574393.861059:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574393.861063:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6144 00000100:00080000:3.0:1788574393.861065:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574393.861069:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574393.861073:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574393.861083:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574393.861083:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864183296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574393.861126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574393.861150:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574393.861164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574394.308868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574394.372883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574395.332870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574395.396803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574396.356834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574396.420805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574396.932953:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574396.932992:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574396.933010:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574396.933048:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574396.933059:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574396.933070:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574396.933081:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788574396.933103:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff92fc6733bb80 x1875449864183040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574396.933139:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864183040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574396.933152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574396.933157:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574396.933161:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574396.933168:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff92fc6733b100 x1875449864183168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574396.933182:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864183168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574396.933189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574396.933193:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574396.933196:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574396.933203:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff92fc67339500 x1875449864183296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574396.933213:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864183296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574396.933221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574396.933223:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574396.933227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183296:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574396.933234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff92fc6733aa00 x1875449864182784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574396.933253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864182784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574396.933261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574396.933264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574396.933267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864182784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574396.933274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574396] req@ffff92fc6733b480 x1875449864182912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574396.933288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864182912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574396.933297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574396.933301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574396.933305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864182912:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788574397.380790:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574397.444805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574398.404794:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574398.468839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574398.980840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574398.980848:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574398.980852:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574398.980861:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6149 00000100:00080000:3.0:1788574398.980870:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574398.980876:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574398.980906:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864183424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574398.980929:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574398.980932:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574398.980934:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574398.980937:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6149 00000100:00080000:3.0:1788574398.980941:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574398.980944:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574398.980955:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574398.980970:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864183552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574398.980980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864183424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574398.980986:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574398.980992:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574398.980994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574398.980999:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6149 00000100:00080000:3.0:1788574398.981005:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574398.981009:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574398.981016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574398.981029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864183680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574398.981046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574398.981049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574398.981054:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574398.981057:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574398.981062:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6149 00000100:00080000:3.0:1788574398.981068:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574398.981073:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574398.981078:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574398.981098:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff92fc67338380 x1875449864183424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.981098:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864183808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574398.981131:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574398.981135:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574398.981137:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574398.981143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864183424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.981144:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6149 00000100:00080000:3.0:1788574398.981151:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574398.981160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788574398.981168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574398.981168:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574398.981177:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574398.981183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864183936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574398.981185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864183424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574398.981223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574398.981251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574398.981266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff92fc67338a80 x1875449864183552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574398.981278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864183552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574398.981286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574398.981289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574398.981293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574398.981299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574398.981318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574398.981329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff92fc6733b800 x1875449864183680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574398.981342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864183680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574398.981350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574398.981353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574398.981356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574398.981361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574398.981382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574398.981393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff92fc67339880 x1875449864183808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574398.981405:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864183808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574398.981412:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574398.981415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574398.981419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574398.981425:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574398.981444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574398.981454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff92fc67339c00 x1875449864183936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574398.981466:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864183936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574398.981473:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574398.981476:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574398.981479:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864183936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574399.428866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574399.492810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574400.452854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574400.516839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574401.476872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574401.540799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574402.500865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574402.564848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574403.524864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574403.588800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574404.100832:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574404.100840:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574404.100844:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574404.100852:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6154 00000100:00080000:2.0:1788574404.100861:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574404.100867:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574404.100899:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864184064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574404.100923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574404.100926:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574404.100927:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574404.100930:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6154 00000100:00080000:2.0:1788574404.100932:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574404.100935:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574404.100950:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574404.100960:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864184192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574404.100968:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574404.100971:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574404.100972:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574404.100978:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6154 00000100:00100000:0.0:1788574404.100980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864184064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574404.100981:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574404.100985:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574404.100991:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574404.100999:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864184320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574404.101005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574404.101007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574404.101009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574404.101011:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6154 00000100:00080000:2.0:1788574404.101014:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574404.101016:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574404.101019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574404.101028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864184448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574404.101034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574404.101037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574404.101038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574404.101040:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6154 00000100:00080000:2.0:1788574404.101042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574404.101044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574404.101048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574404.101055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864184576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574404.101063:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574404.101110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574404/real 1788574404] req@ffff92fc4617e300 x1875449864184064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574404.101131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864184064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574404.101143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574404.101146:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574404.101149:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864184064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574404.101170:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574404.101189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574404.101202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574404/real 1788574404] req@ffff92fc4617d880 x1875449864184192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574404.101215:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864184192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574404.101224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574404.101227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574404.101231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574404.101238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574404.101257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574404.101269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574404/real 1788574404] req@ffff92fc4617d500 x1875449864184320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574404.101288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864184320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574404.101297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574404.101301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574404.101304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574404.101311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574404.101331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574404.101340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574404/real 1788574404] req@ffff92fc4617ea00 x1875449864184448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574404.101348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864184448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574404.101353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574404.101355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574404.101357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574404.101361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574404.101372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574404.101378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574404/real 1788574404] req@ffff92fc4617ce00 x1875449864184576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574404.101385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864184576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574404.101395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574404.101397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574404.101399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574404.548877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574404.612834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574405.572952:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574405.636852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574406.596853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574406.660873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574407.620950:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574407.684876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574408.644821:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574408.708788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574409.220893:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574409.220905:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574409.220910:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574409.220922:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6159 00000100:00080000:2.0:1788574409.220936:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574409.220944:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574409.220990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864184704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574409.221023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574409.221026:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574409.221028:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574409.221031:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6159 00000100:00080000:2.0:1788574409.221033:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574409.221037:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574409.221048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574409.221059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864184832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574409.221067:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574409.221070:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574409.221071:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574409.221074:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6159 00000100:00080000:2.0:1788574409.221076:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574409.221078:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574409.221083:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574409.221085:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864184704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574409.221093:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864184960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574409.221101:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574409.221104:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574409.221105:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574409.221107:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6159 00000100:00080000:2.0:1788574409.221110:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574409.221112:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574409.221115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574409.221124:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864185088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574409.221131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574409.221134:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574409.221136:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574409.221139:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6159 00000100:00080000:2.0:1788574409.221141:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574409.221144:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574409.221146:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574409.221154:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864185216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574409.221165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574409.221215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff92fc4617c000 x1875449864184704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574409.221232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864184704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574409.221241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574409.221244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574409.221247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864184704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574409.221252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574409.221269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574409.221278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff92fc4617fb80 x1875449864184832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574409.221286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864184832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574409.221293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574409.221295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574409.221297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574409.221323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574409.221334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574409.221341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff92fc4617c380 x1875449864184960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574409.221349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864184960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574409.221355:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574409.221356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574409.221359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864184960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574409.221364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574409.221386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574409.221398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff92fc4617d180 x1875449864185088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574409.221411:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864185088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574409.221420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574409.221423:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574409.221427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574409.221431:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574409.221447:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574409.221455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574409/real 1788574409] req@ffff92fc4617e680 x1875449864185216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574409.221463:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864185216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574409.221468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574409.221470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574409.221472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574409.668864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574409.732828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574410.692845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574410.756806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574411.716872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574411.780824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574412.740879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574412.804795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574413.764882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574413.828852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574414.340899:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574414.340913:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574414.340918:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574414.340930:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6164 00000100:00080000:1.0:1788574414.340943:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574414.340951:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574414.341009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864185344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574414.341043:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574414.341046:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574414.341048:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574414.341051:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6164 00000100:00080000:1.0:1788574414.341054:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574414.341057:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574414.341068:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574414.341080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864185472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574414.341089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574414.341092:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574414.341095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574414.341098:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864185344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574414.341100:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6164 00000100:00080000:1.0:1788574414.341106:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574414.341110:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574414.341113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574414.341121:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864185600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574414.341127:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574414.341130:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574414.341131:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574414.341134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6164 00000100:00080000:1.0:1788574414.341137:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574414.341139:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574414.341142:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574414.341150:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864185728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574414.341155:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574414.341158:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574414.341159:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574414.341161:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6164 00000100:00080000:1.0:1788574414.341166:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574414.341169:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574414.341172:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574414.341183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864185856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574414.341187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574414.341250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff92fc67289c00 x1875449864185344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574414.341273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864185344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574414.341285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574414.341289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574414.341295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864185344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574414.341326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574414.341347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574414.341357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff92fc67288380 x1875449864185472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574414.341366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864185472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574414.341372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574414.341375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574414.341377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574414.341381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574414.341393:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574414.341399:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff92fc6728b100 x1875449864185600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574414.341406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864185600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574414.341412:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574414.341414:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574414.341416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574414.341419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574414.341430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574414.341437:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff92fc67288a80 x1875449864185728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574414.341444:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864185728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574414.341449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574414.341451:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574414.341453:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574414.341459:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574414.341478:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574414.341488:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574414/real 1788574414] req@ffff92fc67288700 x1875449864185856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574414.341501:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864185856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574414.341506:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574414.341509:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574414.341511:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864185856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574414.788940:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574414.852799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574415.812878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574415.876836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574416.836855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574416.900852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574417.860913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574417.924871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574418.884892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574418.948803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574419.460858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574419.460870:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574419.460876:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574419.460887:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6170 00000100:00080000:0.0:1788574419.460897:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574419.460905:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574419.460946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864185984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574419.460973:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574419.460976:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574419.460978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574419.460981:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6170 00000100:00080000:0.0:1788574419.460984:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574419.460988:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574419.460996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574419.461007:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864185984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574419.461015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864186112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574419.461028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574419.461030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574419.461032:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574419.461035:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6170 00000100:00080000:0.0:1788574419.461037:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574419.461040:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574419.461043:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574419.461052:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864186240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574419.461064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574419.461066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574419.461069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574419.461070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574419.461079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6170 00000100:00080000:0.0:1788574419.461083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574419.461088:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574419.461094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574419.461112:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff92fc6704a300 x1875449864185984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574419.461121:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864186368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574419.461131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574419.461133:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864185984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574419.461135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574419.461138:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574419.461146:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6170 00000100:00080000:1.0:1788574419.461151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574419.461152:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574419.461159:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:1.0:1788574419.461160:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:0.0:1788574419.461167:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574419.461167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864185984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574419.461183:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864186496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574419.461196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574419.461215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574419.461224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff92fc6704aa00 x1875449864186112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.461233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864186112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.461239:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.461241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574419.461250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574419.461254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574419.461268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574419.461275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff92fc6704a680 x1875449864186240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.461283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864186240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.461288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.461291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574419.461293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574419.461296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574419.461307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574419.461312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff92fc6704ad80 x1875449864186368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.461320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864186368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.461327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.461330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574419.461333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574419.461339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574419.461359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574419.461371:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574419/real 1788574419] req@ffff92fc67049880 x1875449864186496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574419.461389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864186496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574419.461395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574419.461397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574419.461399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574419.908904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574419.972827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574420.932952:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574420.996792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574421.956859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574422.020861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574422.980934:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574423.044900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574424.004928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574424.068921:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574424.580923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574424.580937:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574424.580943:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574424.580955:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6175 00000100:00080000:2.0:1788574424.580969:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574424.580980:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574424.581028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864186624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574424.581064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574424.581068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574424.581070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574424.581074:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6175 00000100:00080000:2.0:1788574424.581079:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574424.581085:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574424.581099:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574424.581108:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864186624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574424.581114:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864186752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574424.581127:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574424.581130:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574424.581132:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574424.581137:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6175 00000100:00080000:2.0:1788574424.581142:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574424.581146:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574424.581152:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574424.581165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864186880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574424.581174:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574424.581179:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574424.581181:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574424.581185:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6175 00000100:00080000:2.0:1788574424.581189:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574424.581195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574424.581199:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574424.581207:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574424.581223:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864187008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574424.581234:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574424.581237:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574424.581239:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574424.581243:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6175 00000100:00100000:0.0:1788574424.581247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff92fc4617dc00 x1875449864186624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574424.581249:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574424.581255:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574424.581259:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574424.581273:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864187136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574424.581281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864186624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.581292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.581296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574424.581306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864186624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574424.581333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574424.581352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574424.581361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff92fc4617ca80 x1875449864186752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.581370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864186752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.581377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.581380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574424.581382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574424.581386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574424.581398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574424.581404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff92fc4617c700 x1875449864186880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.581417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864186880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.581424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.581426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574424.581428:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864186880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574424.581432:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574424.581445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574424.581451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff92fc4617f800 x1875449864187008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.581459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864187008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.581464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.581466:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574424.581468:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574424.581471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574424.581484:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574424.581490:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff92fc4617f100 x1875449864187136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.581498:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864187136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.581509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.581511:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574424.581514:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574425.028985:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574425.092888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574426.052922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574426.116873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574427.076893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574427.140826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574428.100865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574428.164890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574429.124927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574429.188876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574429.700771:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574429.700780:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574429.700784:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574429.700791:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6180 00000100:00080000:2.0:1788574429.700806:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574429.700812:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574429.700843:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864187264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574429.700887:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574429.700891:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574429.700893:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574429.700896:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6180 00000100:00080000:2.0:1788574429.700898:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574429.700901:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574429.700909:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574429.700920:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864187392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574429.700929:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574429.700934:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574429.700936:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574429.700939:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6180 00000100:00080000:2.0:1788574429.700941:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574429.700943:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574429.700947:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574429.700955:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864187520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574429.700962:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574429.700964:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574429.700967:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574429.700974:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864187264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574429.700975:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6180 00000100:00080000:2.0:1788574429.700982:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574429.700995:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574429.700999:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574429.701009:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864187648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574429.701016:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574429.701020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574429.701021:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574429.701024:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6180 00000100:00080000:2.0:1788574429.701026:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574429.701028:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574429.701032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574429.701043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864187776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574429.701078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574429.701127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff92fc4617f480 x1875449864187264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.701144:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864187264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.701155:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.701158:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574429.701162:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864187264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574429.701175:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574429.701199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574429.701216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff92fc4617ed80 x1875449864187392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.701231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864187392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.701240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.701244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574429.701248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574429.701280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574429.701297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574429.701306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff92fc4617f100 x1875449864187520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.701315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864187520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.701321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.701323:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574429.701325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574429.701329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574429.701341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574429.701347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff92fc4617f800 x1875449864187648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.701362:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864187648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.701369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.701372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574429.701374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574429.701378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574429.701391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574429.701398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff92fc4617c700 x1875449864187776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.701405:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864187776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.701411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.701413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574429.701415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864187776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574430.148976:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574430.212880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574431.172895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574431.236826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574432.196945:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574432.260816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574433.220877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574433.284831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574434.244971:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574434.308824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574434.820819:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.820828:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574434.820832:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.820840:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6185 00000100:00080000:2.0:1788574434.820849:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.820855:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574434.820886:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864187904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574434.820910:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.820913:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574434.820926:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.820931:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6185 00000100:00080000:2.0:1788574434.820933:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.820936:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574434.820944:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574434.820955:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864188032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574434.820970:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.820973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574434.820976:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.820981:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6185 00000100:00080000:2.0:1788574434.820985:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.820989:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574434.820995:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574434.821009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864188160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574434.821020:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864187904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574434.821034:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.821043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574434.821045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.821049:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6185 00000100:00080000:2.0:1788574434.821052:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.821054:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574434.821060:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574434.821073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864188288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574434.821080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574434.821083:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574434.821084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574434.821087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6185 00000100:00080000:2.0:1788574434.821089:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574434.821091:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574434.821094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574434.821103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864188416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574434.821137:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574434.821199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff92fc4617ca80 x1875449864187904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.821218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864187904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.821234:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.821238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574434.821242:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864187904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574434.821249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574434.821277:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574434.821293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff92fc4617dc00 x1875449864188032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.821307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864188032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.821317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.821320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574434.821323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574434.821352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574434.821369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574434.821378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff92fc4617e680 x1875449864188160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.821391:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864188160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.821397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.821399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574434.821402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574434.821406:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574434.821417:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574434.821423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff92fc4617d180 x1875449864188288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.821431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864188288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.821436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.821438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574434.821440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574434.821444:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574434.821456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574434.821462:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff92fc4617c380 x1875449864188416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.821470:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864188416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.821480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.821482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574434.821485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574435.268906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574435.332825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574436.292817:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574436.356787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574437.316830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574437.380843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574438.340924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574438.404800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574439.364898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574439.428863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574439.940905:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.940920:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574439.940926:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.940941:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6190 00000100:00080000:2.0:1788574439.940956:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.940965:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574439.941034:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864188544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574439.941070:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.941075:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574439.941077:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.941081:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6190 00000100:00080000:2.0:1788574439.941086:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.941092:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574439.941104:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574439.941120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864188672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574439.941133:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864188544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574439.941135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.941145:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574439.941147:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.941152:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6190 00000100:00080000:2.0:1788574439.941156:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.941160:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574439.941167:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574439.941180:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864188800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574439.941191:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.941195:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574439.941197:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.941202:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6190 00000100:00080000:2.0:1788574439.941206:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.941211:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574439.941217:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574439.941229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574439.941232:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864188928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574439.941244:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574439.941247:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574439.941249:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574439.941254:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6190 00000100:00080000:2.0:1788574439.941258:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574439.941263:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574439.941269:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574439.941282:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864189056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574439.941306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff92fc4617fb80 x1875449864188544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.941334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864188544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.941347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.941352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574439.941357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864188544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574439.941383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574439.941404:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574439.941415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff92fc4617c000 x1875449864188672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.941424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864188672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.941431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.941433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574439.941436:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574439.941439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574439.941451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574439.941457:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff92fc4617ce00 x1875449864188800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.941464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864188800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.941470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.941472:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574439.941474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574439.941477:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574439.941489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574439.941495:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff92fc4617ea00 x1875449864188928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.941503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864188928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.941509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.941511:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574439.941513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864188928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574439.941516:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574439.941527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574439.941532:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff92fc4617d500 x1875449864189056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.941539:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864189056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.941544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.941546:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574439.941549:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574440.388853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574440.452792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574441.412854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574441.476820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574442.436863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574442.500832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574443.460844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574443.524836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574444.484954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574444.548863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574445.060865:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574445.060880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574445.060886:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574445.060897:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6195 00000100:00080000:1.0:1788574445.060909:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574445.060917:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574445.060963:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864189184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574445.060994:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574445.060999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574445.061001:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574445.061006:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6195 00000100:00080000:1.0:1788574445.061010:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574445.061016:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574445.061040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574445.061054:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864189312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574445.061064:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864189184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574445.061073:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574445.061078:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574445.061079:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574445.061083:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6195 00000100:00080000:1.0:1788574445.061086:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574445.061088:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574445.061092:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574445.061101:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864189440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574445.061108:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574445.061111:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574445.061112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574445.061115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6195 00000100:00080000:1.0:1788574445.061129:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574445.061132:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574445.061135:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574445.061141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574445.061147:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864189568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574445.061154:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574445.061157:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574445.061159:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574445.061162:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6195 00000100:00080000:1.0:1788574445.061166:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574445.061168:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574445.061171:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574445.061187:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864189696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574445.061191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574445/real 1788574445] req@ffff92fc67289180 x1875449864189184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574445.061226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864189184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574445.061237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574445.061242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574445.061247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864189184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574445.061279:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574445.061307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574445.061322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574445/real 1788574445] req@ffff92fc67289500 x1875449864189312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574445.061336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864189312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574445.061346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574445.061351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574445.061354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574445.061361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574445.061382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574445.061392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574445/real 1788574445] req@ffff92fc6728bb80 x1875449864189440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574445.061405:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864189440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574445.061411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574445.061413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574445.061415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574445.061419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574445.061432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574445.061439:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574445/real 1788574445] req@ffff92fc67288700 x1875449864189568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574445.061447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864189568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574445.061452:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574445.061454:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574445.061456:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574445.061459:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574445.061470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574445.061476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574445/real 1788574445] req@ffff92fc67288a80 x1875449864189696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574445.061484:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864189696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574445.061494:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574445.061496:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574445.061498:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574445.508893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574445.572893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574446.532851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574446.596841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574447.556842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574447.620844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574448.580886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574448.644839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574449.604840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574449.668802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574450.180894:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574450.180903:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574450.180907:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574450.180916:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6200 00000100:00080000:2.0:1788574450.180926:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574450.180932:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574450.180966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864189824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574450.180992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574450.180995:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574450.180996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574450.180999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6200 00000100:00080000:2.0:1788574450.181001:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574450.181004:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574450.181012:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574450.181022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864189952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574450.181030:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574450.181034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574450.181036:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864189824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574450.181037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574450.181044:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6200 00000100:00080000:2.0:1788574450.181047:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574450.181049:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574450.181052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574450.181062:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864190080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574450.181073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574450.181076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574450.181078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574450.181080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6200 00000100:00080000:2.0:1788574450.181082:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574450.181084:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574450.181105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574450.181110:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574450.181116:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864190208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574450.181125:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574450.181127:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574450.181130:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574450.181134:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6200 00000100:00080000:2.0:1788574450.181138:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574450.181143:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574450.181149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574450.181158:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff92fc4617d880 x1875449864189824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574450.181164:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864190336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574450.181179:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864189824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574450.181189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574450.181192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574450.181195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864189824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574450.181217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574450.181233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574450.181242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff92fc4617e300 x1875449864189952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574450.181250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864189952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574450.181256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574450.181258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574450.181261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864189952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574450.181264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574450.181275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574450.181281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff92fc4617d500 x1875449864190080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574450.181288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864190080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574450.181293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574450.181295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574450.181297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574450.181300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574450.181311:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574450.181317:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff92fc4617ea00 x1875449864190208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574450.181324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864190208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574450.181330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574450.181332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574450.181334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574450.181337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574450.181347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574450.181352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574450/real 1788574450] req@ffff92fc4617ce00 x1875449864190336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574450.181359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864190336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574450.181364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574450.181366:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574450.181368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574450.628865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574450.692822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574451.652856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574451.716836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574452.676874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574452.740849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574453.700886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574453.764814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574454.724868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574454.788834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574455.300905:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574455.300919:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574455.300924:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574455.300938:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6205 00000100:00080000:1.0:1788574455.300952:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574455.300961:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574455.301016:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864190464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574455.301047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574455.301051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574455.301052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574455.301055:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6205 00000100:00080000:1.0:1788574455.301059:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574455.301062:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574455.301075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574455.301092:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864190592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574455.301105:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574455.301110:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864190464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574455.301120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574455.301126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574455.301134:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6205 00000100:00080000:1.0:1788574455.301138:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574455.301156:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574455.301162:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574455.301181:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864190720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574455.301196:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574455.301200:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574455.301202:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574455.301206:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6205 00000100:00080000:1.0:1788574455.301209:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574455.301217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574455.301221:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574455.301227:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574455.301244:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864190848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574455.301254:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574455.301258:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574455.301261:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574455.301267:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6205 00000100:00080000:1.0:1788574455.301273:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574455.301275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff92fc6728b100 x1875449864190464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574455.301282:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574455.301296:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574455.301314:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864190976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574455.301318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864190464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574455.301352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574455.301357:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574455.301361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864190464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574455.301391:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574455.301420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574455.301442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff92fc67288380 x1875449864190592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574455.301457:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864190592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574455.301463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574455.301466:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574455.301468:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574455.301473:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574455.301489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574455.301496:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff92fc67289c00 x1875449864190720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574455.301508:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864190720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574455.301514:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574455.301515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574455.301517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574455.301521:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574455.301532:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574455.301538:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff92fc6728a680 x1875449864190848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574455.301546:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864190848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574455.301550:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574455.301552:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574455.301554:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574455.301558:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574455.301568:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574455.301574:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574455/real 1788574455] req@ffff92fc6728b800 x1875449864190976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574455.301581:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864190976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574455.301591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574455.301593:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574455.301595:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864190976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574455.748862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574455.812809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574456.772872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574456.836835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574457.796893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574457.860802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574458.820812:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574458.884832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574459.844883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574459.908793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574460.420862:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.420869:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574460.420874:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.420882:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6211 00000100:00080000:2.0:1788574460.420892:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.420899:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574460.420944:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864191104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.420976:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.420979:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574460.420981:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.420984:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6211 00000100:00080000:2.0:1788574460.420987:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.420991:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574460.420999:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574460.421011:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864191232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.421019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.421022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574460.421024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.421027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6211 00000100:00080000:2.0:1788574460.421032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574460.421034:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864191104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574460.421038:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574460.421053:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574460.421065:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864191360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.421074:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.421077:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574460.421079:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.421081:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6211 00000100:00080000:2.0:1788574460.421093:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.421095:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574460.421099:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574460.421107:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864191488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574460.421113:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574460.421116:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574460.421117:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574460.421119:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6211 00000100:00080000:2.0:1788574460.421121:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574460.421123:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574460.421126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574460.421133:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864191616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574460.421259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574460.421417:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574460.421450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574460.421478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574460.868892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574460.932838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574461.892890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574461.956847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574462.916888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574462.980896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788574463.493009:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788574463.493057:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574463.493073:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574463.493130:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574463.493141:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574463.493151:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574463.493159:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574463.493287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff92fc4617c380 x1875449864191360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.493316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864191360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.493329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.493334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574463.493339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574463.493348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff92fc4617d180 x1875449864191488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.493361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864191488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.493370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.493374:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574463.493378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574463.493385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff92fc4617e680 x1875449864191616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.493399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864191616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.493407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.493411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574463.493415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191616:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574463.493422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff92fc4617c000 x1875449864191104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.493441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864191104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.493449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.493453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574463.493457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864191104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574463.493463:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574460/real 1788574463] req@ffff92fc4617fb80 x1875449864191232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574463.493475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864191232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574463.493483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574463.493486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574463.493489:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191232:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788574463.940949:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574464.004924:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574464.964888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574465.028901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574465.540864:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.540875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574465.540879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.540889:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6216 00000100:00080000:2.0:1788574465.540900:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.540909:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574465.540958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864191744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574465.540986:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.540990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574465.540992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.540996:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6216 00000100:00080000:2.0:1788574465.541000:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.541005:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574465.541016:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574465.541032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864191872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574465.541033:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864191744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574465.541051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.541055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574465.541058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.541062:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6216 00000100:00080000:2.0:1788574465.541066:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.541070:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574465.541074:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574465.541085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864192000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574465.541097:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574465.541101:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574465.541103:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574465.541106:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574465.541111:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6216 00000100:00080000:2.0:1788574465.541116:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.541121:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574465.541126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574465.541137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864192128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574465.541147:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574465.541151:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574465.541156:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574465.541156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff92fc4617dc00 x1875449864191744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574465.541165:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6216 00000100:00080000:2.0:1788574465.541172:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574465.541177:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574465.541181:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574465.541187:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864191744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574465.541194:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864192256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574465.541207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.541219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574465.541224:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864191744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574465.541252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574465.541295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574465.541311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff92fc4617ca80 x1875449864191872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.541324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864191872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.541332:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.541335:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574465.541339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864191872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574465.541345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574465.541363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574465.541375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff92fc4617c700 x1875449864192000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.541386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864192000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.541394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.541397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574465.541399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574465.541405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574465.541433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574465.541444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff92fc4617f800 x1875449864192128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.541456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864192128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.541464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.541467:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574465.541470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574465.541476:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574465.541495:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574465.541506:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574465/real 1788574465] req@ffff92fc4617f100 x1875449864192256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574465.541518:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864192256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574465.541527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574465.541530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574465.541534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574465.988888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574466.052838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574467.012868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574467.076826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574468.036910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574468.100838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574469.060899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574469.124828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574470.084902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574470.148834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574470.660801:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.660809:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574470.660813:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.660821:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6221 00000100:00080000:1.0:1788574470.660829:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.660836:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574470.660868:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864192384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.660889:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.660892:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574470.660894:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.660896:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6221 00000100:00080000:1.0:1788574470.660898:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.660900:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574470.660908:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574470.660917:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864192512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.660924:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.660927:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574470.660928:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.660930:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6221 00000100:00080000:1.0:1788574470.660933:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.660935:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574470.660939:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574470.660944:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864192384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574470.660948:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864192640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.660959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.660962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574470.660963:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.660966:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6221 00000100:00080000:1.0:1788574470.660969:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.660971:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574470.660974:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574470.660982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864192768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.660988:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.660990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574470.660991:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.660994:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6221 00000100:00080000:1.0:1788574470.660997:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.660999:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574470.661002:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574470.661011:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864192896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574470.661018:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574470.661060:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff92fc6728ad80 x1875449864192384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.661080:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864192384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.661112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.661116:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574470.661120:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864192384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574470.661128:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574470.661150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574470.661163:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff92fc67289880 x1875449864192512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.661176:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864192512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.661185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.661189:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574470.661193:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574470.661217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574470.661235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574470.661245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff92fc67289f80 x1875449864192640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.661257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864192640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.661266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.661270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574470.661273:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574470.661280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574470.661297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574470.661306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff92fc67288000 x1875449864192768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.661337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864192768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.661347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.661350:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574470.661353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574470.661359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574470.661378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574470.661389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff92fc67288e00 x1875449864192896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574470.661401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864192896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574470.661409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574470.661412:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574470.661416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864192896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574471.108975:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574471.172849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574472.132915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574472.196871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574473.156884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574473.220807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574474.180903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574474.244847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574475.204897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574475.268844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574475.780840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574475.780850:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574475.780854:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.780865:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6226 00000100:00080000:1.0:1788574475.780876:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.780883:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574475.780921:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864193024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.780962:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574475.780966:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574475.780968:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.780972:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6226 00000100:00080000:1.0:1788574475.780976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.780980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574475.780989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574475.781003:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864193152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.781015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574475.781017:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864193024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574475.781019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574475.781024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.781030:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6226 00000100:00080000:1.0:1788574475.781034:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.781037:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574475.781043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574475.781056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864193280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.781066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574475.781069:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574475.781071:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.781075:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6226 00000800:00000100:3.0:1788574475.781079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574475.781082:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.781089:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574475.781095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574475.781107:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864193408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.781116:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574475.781120:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff92fc6728a300 x1875449864193024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574475.781122:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574475.781127:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.781137:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6226 00000100:00080000:1.0:1788574475.781142:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574475.781143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864193024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574475.781150:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:3.0:1788574475.781156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788574475.781159:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574475.781161:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574475.781168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864193024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574475.781178:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864193536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574475.781188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574475.781206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574475.781216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff92fc6728b480 x1875449864193152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574475.781225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864193152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574475.781231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574475.781233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574475.781236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574475.781239:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574475.781249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574475.781255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff92fc6728aa00 x1875449864193280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574475.781262:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864193280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574475.781272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574475.781273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574475.781275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574475.781278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574475.781288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574475.781294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff92fc67288e00 x1875449864193408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574475.781301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864193408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574475.781305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574475.781307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574475.781309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574475.781318:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574475.781327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574475.781334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574475] req@ffff92fc67288000 x1875449864193536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574475.781341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864193536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574475.781345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574475.781347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574475.781350:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574476.228876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574476.292831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574477.252864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574477.316815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574478.276881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574478.340854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574479.300885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574479.364839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574480.324903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574480.388866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574480.900896:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574480.900912:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574480.900919:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574480.900944:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6231 00000100:00080000:1.0:1788574480.900958:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574480.900967:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574480.901012:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864193664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574480.901040:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574480.901043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574480.901045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574480.901053:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6231 00000100:00080000:1.0:1788574480.901056:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574480.901059:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574480.901070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574480.901080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864193792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574480.901088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574480.901091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574480.901095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574480.901097:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864193664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574480.901099:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6231 00000100:00080000:1.0:1788574480.901106:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574480.901109:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574480.901112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574480.901121:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864193920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574480.901132:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574480.901135:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574480.901137:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574480.901139:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6231 00000100:00080000:1.0:1788574480.901142:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574480.901144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574480.901148:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574480.901155:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864194048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574480.901161:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574480.901164:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574480.901166:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574480.901168:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6231 00000100:00080000:1.0:1788574480.901171:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574480.901173:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574480.901177:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574480.901181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574480.901188:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864194176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574480.901230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff92fc67289f80 x1875449864193664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.901246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864193664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.901255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.901259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574480.901262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864193664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574480.901285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574480.901301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574480.901310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff92fc67289880 x1875449864193792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.901319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864193792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.901325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.901327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574480.901329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574480.901333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574480.901343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574480.901350:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff92fc6728ad80 x1875449864193920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.901358:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864193920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.901374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.901376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574480.901379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864193920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574480.901383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574480.901396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574480.901403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff92fc6728b800 x1875449864194048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.901411:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864194048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.901416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.901419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574480.901421:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574480.901424:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574480.901453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574480.901461:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff92fc6728a680 x1875449864194176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.901469:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864194176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.901475:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.901477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574480.901479:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574481.348922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574481.412873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574482.372827:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574482.436817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574483.396870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574483.460824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574484.420848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574484.484843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574485.444870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574485.508874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574486.020829:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574486.020841:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574486.020846:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574486.020857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6236 00000100:00080000:2.0:1788574486.020867:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574486.020875:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574486.020914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864194304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574486.020947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574486.020954:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574486.020956:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574486.020960:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6236 00000100:00080000:2.0:1788574486.020963:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574486.020968:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574486.020979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574486.020995:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864194304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574486.020996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864194432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574486.021013:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574486.021016:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574486.021018:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574486.021022:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6236 00000100:00080000:2.0:1788574486.021026:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574486.021031:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574486.021037:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574486.021052:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864194560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574486.021058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574486.021069:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574486.021073:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574486.021075:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574486.021080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6236 00000100:00080000:2.0:1788574486.021085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574486.021089:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574486.021097:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574486.021097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574486/real 1788574486] req@ffff92fc4617ed80 x1875449864194304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574486.021114:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864194688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574486.021117:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864194304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574486.021146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574486.021147:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574486.021152:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788574486.021156:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788574486.021158:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864194304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574486.021160:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574486.021167:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6236 00000100:00080000:2.0:1788574486.021172:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574486.021177:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574486.021182:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194432:192.168.202.132@tcp:38:kworker.0 00000008:00100000:2.0:1788574486.021183:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574486.021204:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864194816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574486.021209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574486.021223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574486/real 1788574486] req@ffff92fc4617f480 x1875449864194432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574486.021233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864194432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574486.021240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574486.021242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574486.021245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574486.021250:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574486.021260:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574486.021267:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574486/real 1788574486] req@ffff92fc4617f100 x1875449864194560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574486.021274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864194560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574486.021279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574486.021282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574486.021284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574486.021287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574486.021297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574486.021302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574486/real 1788574486] req@ffff92fc4617f800 x1875449864194688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574486.021309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864194688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574486.021314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574486.021316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574486.021317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574486.021326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574486.021337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574486.021343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574486/real 1788574486] req@ffff92fc4617c700 x1875449864194816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574486.021351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864194816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574486.021356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574486.021358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574486.021359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864194816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574486.468812:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574486.532852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574487.492864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574487.556833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574488.516839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574488.580880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574489.540957:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574489.604912:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574490.564924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574490.628871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574491.140875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574491.140887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574491.140892:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574491.140907:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6241 00000100:00080000:1.0:1788574491.140923:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574491.140932:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574491.140992:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864194944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574491.141018:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574491.141021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574491.141023:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574491.141026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6241 00000100:00080000:1.0:1788574491.141030:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574491.141034:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574491.141047:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574491.141068:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864195072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574491.141081:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574491.141086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574491.141087:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864194944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574491.141089:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574491.141097:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6241 00000100:00080000:1.0:1788574491.141101:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574491.141104:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574491.141109:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574491.141124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864195200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574491.141134:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574491.141139:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574491.141142:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574491.141146:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6241 00000100:00080000:1.0:1788574491.141160:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574491.141166:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574491.141170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788574491.141174:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574491.141187:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864195328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574491.141195:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574491.141198:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574491.141200:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574491.141203:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6241 00000100:00080000:1.0:1788574491.141206:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574491.141209:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574491.141212:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574491.141226:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864195456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574491.141226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574491/real 1788574491] req@ffff92fc67289c00 x1875449864194944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574491.141268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864194944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574491.141279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574491.141283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574491.141287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864194944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574491.141310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574491.141326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574491.141335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574491/real 1788574491] req@ffff92fc67288380 x1875449864195072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574491.141343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864195072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574491.141354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574491.141356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574491.141358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574491.141362:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574491.141372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574491.141378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574491/real 1788574491] req@ffff92fc6728b100 x1875449864195200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574491.141385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864195200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574491.141391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574491.141393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574491.141394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574491.141398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574491.141409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574491.141415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574491/real 1788574491] req@ffff92fc67288a80 x1875449864195328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574491.141423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864195328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574491.141427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574491.141429:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574491.141432:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574491.141435:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574491.141445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574491.141450:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574491/real 1788574491] req@ffff92fc67288700 x1875449864195456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574491.141456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864195456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574491.141461:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574491.141463:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574491.141465:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574491.588893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574491.652885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574492.612855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574492.676801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574493.636852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574493.700840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574494.660852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574494.724798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574495.684850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574495.748826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574496.260829:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574496.260841:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574496.260846:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574496.260856:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6246 00000100:00080000:1.0:1788574496.260868:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574496.260876:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574496.260925:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864195584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574496.260957:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574496.260962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574496.260965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574496.260980:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6246 00000100:00080000:1.0:1788574496.260988:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574496.260995:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574496.261008:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574496.261030:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864195584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574496.261031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864195712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574496.261059:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574496.261063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574496.261067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574496.261071:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6246 00000100:00080000:1.0:1788574496.261076:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574496.261081:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574496.261088:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574496.261104:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864195840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574496.261116:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574496.261120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574496.261123:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574496.261128:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6246 00000100:00080000:1.0:1788574496.261133:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574496.261139:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574496.261147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574496.261173:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864195968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574496.261185:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574496.261188:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574496.261191:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574496.261196:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6246 00000100:00080000:1.0:1788574496.261201:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574496.261206:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574496.261213:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574496.261227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864196096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574496.261398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574496.261446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff92fc6728bb80 x1875449864195584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.261464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864195584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.261474:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.261477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574496.261480:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864195584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574496.261506:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574496.261523:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574496.261534:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff92fc67289500 x1875449864195712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.261543:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864195712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.261549:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.261552:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574496.261555:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574496.261559:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574496.261589:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574496.261598:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff92fc67289180 x1875449864195840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.261606:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864195840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.261612:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.261616:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574496.261619:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574496.261626:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574496.261648:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574496.261661:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff92fc67288700 x1875449864195968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.261673:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864195968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.261680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.261682:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574496.261685:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864195968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574496.261689:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574496.261704:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574496.261711:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574496/real 1788574496] req@ffff92fc67288a80 x1875449864196096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574496.261719:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864196096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574496.261766:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574496.261769:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574496.261772:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574496.708913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574496.772837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574497.732903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574497.796882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574498.756972:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574498.820821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574499.780932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574499.844901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574500.804905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574500.868877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574501.380826:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574501.380838:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574501.380842:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574501.380853:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6251 00000100:00080000:1.0:1788574501.380865:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574501.380873:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574501.380918:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864196224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574501.380945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574501.380951:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574501.380952:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574501.380955:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6251 00000100:00080000:1.0:1788574501.380958:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574501.380961:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574501.380973:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574501.380990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864196352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574501.380993:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864196224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574501.381013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574501.381017:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574501.381019:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574501.381024:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6251 00000100:00080000:1.0:1788574501.381028:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574501.381033:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574501.381039:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574501.381054:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864196480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574501.381074:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574501.381080:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574501.381081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574501.381084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574501.381091:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6251 00000100:00080000:1.0:1788574501.381097:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574501.381102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574501.381110:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574501.381124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864196608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574501.381136:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574501.381141:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788574501.381143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff92fc6728b100 x1875449864196224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574501.381145:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574501.381154:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6251 00000100:00080000:1.0:1788574501.381162:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574501.381169:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574501.381175:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574501.381179:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864196224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574501.381192:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864196736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574501.381196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.381208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574501.381215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864196224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574501.381248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574501.381282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574501.381294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff92fc67288380 x1875449864196352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.381303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864196352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.381309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.381312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574501.381314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574501.381318:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574501.381331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574501.381338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff92fc67289c00 x1875449864196480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.381345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864196480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.381351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.381352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574501.381354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574501.381358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574501.381369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574501.381375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff92fc6728a680 x1875449864196608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.381383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864196608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.381389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.381390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574501.381392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574501.381395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574501.381405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574501.381411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574501/real 1788574501] req@ffff92fc6728b800 x1875449864196736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574501.381418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864196736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574501.381424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574501.381425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574501.381427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574501.828873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574501.892895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574502.852949:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574502.916888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574503.876923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574503.940876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574504.900906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574504.964883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574505.924813:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574505.988837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574506.500858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574506.500866:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574506.500870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574506.500878:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6257 00000100:00080000:0.0:1788574506.500888:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574506.500895:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574506.500930:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864196864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574506.500957:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574506.500962:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574506.500965:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574506.500969:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6257 00000100:00080000:0.0:1788574506.500973:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574506.500978:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574506.500989:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574506.501004:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864196992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574506.501010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864196864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574506.501034:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574506.501039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574506.501042:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574506.501047:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6257 00000100:00080000:0.0:1788574506.501052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574506.501057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574506.501063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574506.501068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574506.501080:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864197120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574506.501093:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574506.501096:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574506.501099:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574506.501104:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6257 00000100:00080000:0.0:1788574506.501109:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574506.501110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff92fc6704b100 x1875449864196864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574506.501117:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574506.501126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574506.501132:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864196864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574506.501145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864197248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574506.501160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574506.501165:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574506.501168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788574506.501172:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574506.501177:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574506.501177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864196864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574506.501187:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6257 00000100:00080000:0.0:1788574506.501192:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574506.501196:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574506.501202:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574506.501204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574506.501220:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864197376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574506.501230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574506.501245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff92fc67049500 x1875449864196992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574506.501255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864196992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574506.501262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574506.501264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574506.501266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864196992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574506.501270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574506.501281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574506.501288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff92fc67049180 x1875449864197120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574506.501295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864197120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574506.501300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574506.501302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574506.501303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574506.501307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574506.501318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574506.501324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff92fc67049c00 x1875449864197248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574506.501331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864197248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574506.501336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574506.501338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574506.501340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574506.501349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574506.501359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574506.501365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574506/real 1788574506] req@ffff92fc67048e00 x1875449864197376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574506.501376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864197376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574506.501383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574506.501387:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574506.501390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574506.948859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574507.012837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574507.972865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574508.036861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574508.996908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574509.060886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574510.020950:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574510.084848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574511.044897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574511.108882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574511.620902:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574511.620916:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574511.620930:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574511.620946:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6262 00000100:00080000:2.0:1788574511.620958:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574511.620966:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574511.621015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864197504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574511.621054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574511.621059:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574511.621061:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574511.621065:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6262 00000100:00080000:2.0:1788574511.621069:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574511.621072:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574511.621083:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574511.621096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864197632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574511.621106:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574511.621108:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864197504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574511.621111:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574511.621116:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574511.621123:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6262 00000100:00080000:2.0:1788574511.621127:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574511.621131:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574511.621137:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574511.621153:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864197760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574511.621170:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574511.621173:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574511.621175:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574511.621180:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6262 00000100:00080000:2.0:1788574511.621185:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574511.621189:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574511.621195:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574511.621202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574511.621212:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864197888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574511.621229:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574511.621232:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574511.621234:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574511.621237:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6262 00000100:00080000:2.0:1788574511.621240:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574511.621242:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574511.621247:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574511.621257:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864198016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574511.621265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff92fc4617ca80 x1875449864197504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.621293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864197504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.621302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.621305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574511.621309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864197504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574511.621333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574511.621352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574511.621362:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff92fc4617dc00 x1875449864197632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.621370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864197632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.621377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.621379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574511.621381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574511.621384:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574511.621396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574511.621402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff92fc4617fb80 x1875449864197760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.621412:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864197760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.621427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.621431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574511.621434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574511.621441:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574511.621464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574511.621475:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff92fc4617c000 x1875449864197888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.621483:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864197888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.621488:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.621491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574511.621493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864197888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574511.621497:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574511.621510:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574511.621516:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff92fc4617e680 x1875449864198016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574511.621524:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864198016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574511.621529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574511.621530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574511.621533:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574512.068934:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574512.132828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574513.092873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574513.156831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574514.116918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574514.180852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574515.140909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574515.204857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574516.164922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574516.228831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574516.740898:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574516.740913:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574516.740919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574516.740933:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6267 00000100:00080000:2.0:1788574516.740946:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574516.740955:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574516.741016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864198144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574516.741052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574516.741060:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574516.741062:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574516.741065:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6267 00000100:00080000:2.0:1788574516.741069:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574516.741072:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574516.741084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574516.741096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864198272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574516.741105:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864198144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574516.741109:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574516.741113:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574516.741114:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574516.741118:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6267 00000100:00080000:2.0:1788574516.741120:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574516.741123:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574516.741126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574516.741137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864198400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574516.741144:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574516.741147:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574516.741148:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574516.741151:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6267 00000100:00080000:2.0:1788574516.741153:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574516.741155:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574516.741159:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574516.741167:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864198528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574516.741173:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574516.741183:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574516.741185:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574516.741188:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6267 00000100:00080000:2.0:1788574516.741191:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574516.741193:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574516.741197:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574516.741201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574516.741209:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864198656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574516.741263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff92fc4617d180 x1875449864198144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.741285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864198144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.741298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.741303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574516.741308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864198144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574516.741339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574516.741360:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574516.741370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff92fc4617c380 x1875449864198272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.741380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864198272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.741387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.741391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574516.741395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574516.741401:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574516.741435:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574516.741448:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff92fc4617ce00 x1875449864198400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.741461:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864198400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.741470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.741474:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574516.741477:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574516.741484:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574516.741506:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574516.741519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff92fc4617ea00 x1875449864198528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.741532:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864198528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.741541:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.741545:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574516.741549:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574516.741555:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574516.741588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574516.741599:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff92fc4617d500 x1875449864198656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.741607:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864198656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.741614:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.741616:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574516.741618:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574517.188901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574517.252914:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574518.212958:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574518.276880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574519.236866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574519.300898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574520.260902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574520.324839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574521.284877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574521.348892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574521.860875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574521.860891:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574521.860896:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574521.860909:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6272 00000100:00080000:2.0:1788574521.860923:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574521.860932:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574521.860982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864198784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574521.861016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574521.861020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574521.861022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574521.861027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6272 00000100:00080000:2.0:1788574521.861030:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574521.861035:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574521.861047:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574521.861061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864198912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574521.861080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574521.861084:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574521.861086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574521.861092:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6272 00000100:00080000:2.0:1788574521.861096:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574521.861105:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574521.861107:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864198784:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574521.861112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574521.861136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864199040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574521.861150:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574521.861154:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574521.861157:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574521.861163:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6272 00000100:00080000:2.0:1788574521.861169:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574521.861174:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574521.861180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788574521.861183:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574521.861202:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864199168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574521.861216:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574521.861220:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574521.861223:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574521.861226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff92fc4617e300 x1875449864198784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574521.861233:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6272 00000100:00080000:2.0:1788574521.861243:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574521.861254:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574521.861254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864198784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574521.861271:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574521.861272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.861277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574521.861284:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864198784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574521.861292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574521.861299:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864199296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574521.861330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574521.861343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff92fc4617d880 x1875449864198912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.861351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864198912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.861366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.861369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574521.861371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864198912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574521.861398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574521.861412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574521.861419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff92fc4617d500 x1875449864199040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.861427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864199040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.861433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.861435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574521.861437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574521.861441:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574521.861452:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574521.861458:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff92fc4617ea00 x1875449864199168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.861465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864199168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.861471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.861472:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574521.861474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574521.861478:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574521.861489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574521.861495:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff92fc4617ce00 x1875449864199296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.861503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864199296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.861508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.861510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574521.861512:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574522.308937:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574522.372894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574523.332829:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574523.396877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574524.356892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574524.420830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574525.380949:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574525.444877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574526.404906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574526.468869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574526.980874:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574526.980882:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574526.980887:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574526.980895:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6277 00000100:00080000:0.0:1788574526.980904:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574526.980911:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574526.980942:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864199424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574526.980963:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574526.980966:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574526.980968:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574526.980971:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6277 00000100:00080000:0.0:1788574526.980973:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574526.980975:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574526.980982:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574526.980993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864199552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574526.981001:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574526.981004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574526.981006:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574526.981011:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864199424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574526.981012:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6277 00000100:00080000:0.0:1788574526.981015:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574526.981018:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574526.981021:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574526.981029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864199680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574526.981035:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574526.981037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574526.981039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574526.981042:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6277 00000100:00080000:0.0:1788574526.981045:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574526.981047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574526.981051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574526.981059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864199808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574526.981065:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574526.981067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574526.981068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574526.981071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6277 00000100:00080000:0.0:1788574526.981074:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574526.981076:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574526.981079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574526.981085:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864199936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574526.981132:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574526.981164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574526.981185:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574526.981212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574527.428894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574527.492868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574528.452833:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574528.516862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574529.476890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574529.540851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788574530.052989:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788574530.053024:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574530.053037:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574530.053076:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574530.053087:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574530.053096:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574530.053110:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574530.053136:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574530] req@ffff92fc67048e00 x1875449864199552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574530.053168:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864199552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574530.053179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574530.053183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574530.053187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574530.053193:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574530] req@ffff92fc67049c00 x1875449864199680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574530.053207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864199680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574530.053212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574530.053215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574530.053218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574530.053225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574530] req@ffff92fc67049180 x1875449864199808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574530.053238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864199808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574530.053246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574530.053266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574530.053269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574530.053277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574530] req@ffff92fc67049500 x1875449864199936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574530.053292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864199936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574530.053300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574530.053303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574530.053306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864199936:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574530.053314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574530] req@ffff92fc6704bb80 x1875449864199424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574530.053325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864199424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574530.053332:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574530.053336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574530.053339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864199424:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788574530.500923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574530.564845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574531.524966:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574531.588849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574532.100883:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574532.100895:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574532.100900:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574532.100914:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6282 00000100:00080000:2.0:1788574532.100929:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574532.100939:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574532.100988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864200064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574532.101026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574532.101030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574532.101032:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574532.101037:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6282 00000100:00080000:2.0:1788574532.101042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574532.101047:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574532.101061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574532.101079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864200192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574532.101091:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864200064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574532.101096:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574532.101105:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574532.101107:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574532.101112:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6282 00000100:00080000:2.0:1788574532.101116:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574532.101122:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574532.101128:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574532.101143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864200320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574532.101154:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574532.101159:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574532.101161:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574532.101166:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6282 00000100:00080000:2.0:1788574532.101171:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574532.101188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574532.101193:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574532.101202:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574532.101218:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864200448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574532.101228:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574532.101232:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574532.101234:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574532.101239:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6282 00000100:00080000:2.0:1788574532.101243:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574532.101251:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574532.101254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574532/real 1788574532] req@ffff92fc4617c380 x1875449864200064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574532.101260:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574532.101280:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864200576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574532.101286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864200064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574532.101310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574532.101315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574532.101321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864200064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574532.101355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574532.101386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574532.101402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574532/real 1788574532] req@ffff92fc4617d180 x1875449864200192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574532.101415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864200192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574532.101424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574532.101428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574532.101432:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574532.101438:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574532.101462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574532.101474:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574532/real 1788574532] req@ffff92fc4617e680 x1875449864200320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574532.101487:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864200320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574532.101496:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574532.101499:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574532.101502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574532.101509:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574532.102264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574532.102286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574532/real 1788574532] req@ffff92fc4617c000 x1875449864200448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574532.102303:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864200448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574532.102312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574532.102317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574532.102321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574532.102329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574532.102358:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574532.102374:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574532/real 1788574532] req@ffff92fc4617fb80 x1875449864200576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574532.102387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864200576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574532.102395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574532.102399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574532.102402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574532.548913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574532.614661:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574533.572890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574533.636834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574534.596891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574534.660879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574535.620946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574535.684830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574536.644925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574536.708851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574537.220978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.220990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574537.220996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.221010:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6287 00000100:00080000:2.0:1788574537.221025:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.221036:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574537.221089:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864200704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.221115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.221118:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574537.221120:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.221123:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6287 00000100:00080000:2.0:1788574537.221126:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.221129:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574537.221138:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574537.221148:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864200832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.221156:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.221159:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574537.221160:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.221163:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6287 00000100:00080000:2.0:1788574537.221166:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.221169:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574537.221172:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574537.221177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864200704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574537.221183:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864200960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.221191:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.221194:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574537.221195:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.221197:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6287 00000100:00080000:2.0:1788574537.221199:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.221202:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574537.221205:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574537.221213:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864201088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574537.221219:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574537.221222:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574537.221234:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574537.221236:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6287 00000100:00080000:2.0:1788574537.221249:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574537.221252:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574537.221255:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574537.221263:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864201216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574537.221273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574537.221331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff92fc4617dc00 x1875449864200704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574537.221349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864200704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574537.221361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574537.221366:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574537.221370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864200704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574537.221379:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574537.221406:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574537.221418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff92fc4617ca80 x1875449864200832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574537.221432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864200832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574537.221439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574537.221441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574537.221443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574537.221483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574537.221507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574537.221518:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff92fc4617c700 x1875449864200960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574537.221527:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864200960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574537.221533:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574537.221536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574537.221538:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864200960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574537.221543:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574537.221557:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574537.221564:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff92fc4617f800 x1875449864201088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574537.221571:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864201088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574537.221581:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574537.221584:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574537.221586:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574537.221589:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574537.221600:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574537.221606:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574537/real 1788574537] req@ffff92fc4617f100 x1875449864201216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574537.221613:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864201216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574537.221619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574537.221621:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574537.221623:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574537.668933:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574537.732891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574538.692917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574538.756901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574539.716900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574539.780840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574540.740867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574540.804816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574541.764993:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574541.828843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574542.340877:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.340889:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574542.340894:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.340906:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6292 00000100:00080000:2.0:1788574542.340918:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.340926:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574542.340982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864201344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.341006:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.341009:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574542.341011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.341014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6292 00000100:00080000:2.0:1788574542.341016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.341018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574542.341029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574542.341039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864201472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.341048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.341050:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574542.341052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.341056:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6292 00000100:00080000:2.0:1788574542.341058:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.341063:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574542.341069:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574542.341069:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864201344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574542.341082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864201600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.341090:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.341093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574542.341094:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.341100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6292 00000100:00080000:2.0:1788574542.341103:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.341106:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574542.341109:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574542.341117:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864201728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574542.341122:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574542.341125:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574542.341126:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574542.341129:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6292 00000100:00080000:2.0:1788574542.341132:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574542.341135:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574542.341138:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574542.341139:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574542.341152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864201856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574542.341192:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff92fc4617f480 x1875449864201344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574542.341207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864201344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574542.341217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574542.341220:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574542.341223:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864201344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574542.341246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574542.341263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574542.341271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff92fc4617ed80 x1875449864201472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574542.341280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864201472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574542.341285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574542.341288:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574542.341290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574542.341295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574542.341313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574542.341326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff92fc4617f100 x1875449864201600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574542.341354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864201600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574542.341363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574542.341367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574542.341370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574542.341377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574542.341399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574542.341410:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff92fc4617f800 x1875449864201728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574542.341422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864201728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574542.341429:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574542.341432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574542.341435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574542.341440:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574542.341457:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574542.341467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574542/real 1788574542] req@ffff92fc4617c700 x1875449864201856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574542.341477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864201856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574542.341484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574542.341488:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574542.341491:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864201856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574542.788893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574542.852797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574543.812959:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574543.876807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574544.836902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574544.900821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574545.860993:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574545.924801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574546.884853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574546.948791:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574547.460809:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574547.460817:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574547.460821:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574547.460829:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6298 00000100:00080000:1.0:1788574547.460838:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574547.460844:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574547.460890:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864201984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574547.460918:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574547.460921:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574547.460922:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574547.460925:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6298 00000100:00080000:1.0:1788574547.460927:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574547.460930:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574547.460938:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574547.460948:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864202112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574547.460956:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574547.460959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574547.460961:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574547.460963:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6298 00000100:00080000:1.0:1788574547.460965:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574547.460968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574547.460970:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574547.460979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864202240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574547.460987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574547.460989:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574547.460992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574547.461002:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6298 00000100:00100000:3.0:1788574547.461002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864201984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574547.461005:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574547.461012:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574547.461022:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574547.461030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864202368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574547.461036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574547.461038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574547.461040:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574547.461043:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6298 00000100:00080000:1.0:1788574547.461044:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574547.461047:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574547.461051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574547.461058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864202496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574547.461119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574547.461175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff92fc6728ad80 x1875449864201984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574547.461194:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864201984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574547.461210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574547.461214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574547.461218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864201984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574547.461225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574547.461245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574547.461255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff92fc67289880 x1875449864202112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574547.461264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864202112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574547.461293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574547.461295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574547.461298:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574547.461329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574547.461346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574547.461355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff92fc67289f80 x1875449864202240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574547.461365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864202240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574547.461373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574547.461375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574547.461378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574547.461384:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574547.461398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574547.461407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff92fc67288000 x1875449864202368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574547.461417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864202368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574547.461440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574547.461442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574547.461444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574547.461449:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574547.461463:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574547.461470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574547/real 1788574547] req@ffff92fc67288e00 x1875449864202496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574547.461478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864202496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574547.461484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574547.461486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574547.461488:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574547.908950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574547.972908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574548.932945:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574548.996832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574549.956852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574550.020859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574550.980904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574551.044803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574552.004868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574552.068823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574552.580874:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574552.580885:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574552.580890:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574552.580930:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6303 00000100:00080000:2.0:1788574552.580941:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574552.580948:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574552.580987:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864202624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574552.581016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574552.581020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574552.581022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574552.581026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6303 00000100:00080000:2.0:1788574552.581030:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574552.581033:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574552.581042:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574552.581057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864202752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574552.581069:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574552.581073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574552.581077:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574552.581079:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864202624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574552.581084:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6303 00000100:00080000:2.0:1788574552.581090:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574552.581095:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574552.581101:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574552.581115:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864202880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574552.581132:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574552.581136:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574552.581138:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574552.581141:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6303 00000100:00080000:2.0:1788574552.581145:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574552.581151:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574552.581152:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574552.581162:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574552.581175:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864203008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574552.581187:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574552.581192:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574552.581195:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574552.581199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff92fc4617ca80 x1875449864202624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574552.581200:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6303 00000100:00080000:2.0:1788574552.581209:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574552.581216:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574552.581221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864202624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574552.581222:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574552.581235:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574552.581241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574552.581243:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864203136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574552.581245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864202624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574552.581269:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574552.581288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574552.581297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff92fc4617dc00 x1875449864202752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574552.581305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864202752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574552.581311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574552.581313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574552.581315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574552.581319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574552.581329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574552.581335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff92fc4617fb80 x1875449864202880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574552.581342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864202880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574552.581352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574552.581354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574552.581356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864202880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574552.581359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574552.581369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574552.581374:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff92fc4617c000 x1875449864203008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574552.581392:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864203008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574552.581398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574552.581400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574552.581402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574552.581405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574552.581416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574552.581422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff92fc4617e680 x1875449864203136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574552.581429:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864203136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574552.581434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574552.581436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574552.581439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574553.028856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574553.092803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574554.052917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574554.116846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574555.076929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574555.140799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574556.100861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574556.164833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574557.124846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574557.188830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574557.700819:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574557.700831:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574557.700835:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574557.700846:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6308 00000100:00080000:2.0:1788574557.700857:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574557.700864:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574557.700917:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864203264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574557.700941:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574557.700945:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574557.700947:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574557.700951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6308 00000100:00080000:2.0:1788574557.700956:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574557.700960:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574557.700970:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574557.700985:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864203392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574557.700996:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574557.700999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574557.701001:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574557.701005:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6308 00000100:00080000:2.0:1788574557.701008:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574557.701012:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574557.701017:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574557.701025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864203264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574557.701028:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864203520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574557.701037:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574557.701041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574557.701043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574557.701046:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6308 00000100:00080000:2.0:1788574557.701049:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574557.701053:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574557.701058:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574557.701067:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864203648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574557.701075:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574557.701078:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574557.701080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574557.701082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574557.701084:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6308 00000100:00080000:2.0:1788574557.701087:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574557.701090:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574557.701094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574557.701103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864203776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574557.701122:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff92fc4617d180 x1875449864203264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.701142:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864203264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.701152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.701156:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574557.701160:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864203264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574557.701168:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574557.701187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574557.701197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff92fc4617c380 x1875449864203392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.701205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864203392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.701211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.701213:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574557.701215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574557.701233:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574557.701245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574557.701253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff92fc4617ce00 x1875449864203520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.701260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864203520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.701266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.701268:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574557.701270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574557.701273:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574557.701293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574557.701301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff92fc4617ea00 x1875449864203648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.701309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864203648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.701314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.701317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574557.701319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574557.701326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574557.701337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574557.701343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff92fc4617d500 x1875449864203776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574557.701351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864203776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574557.701357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574557.701359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574557.701361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864203776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574558.148937:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574558.212913:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574559.172848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574559.236840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574560.196825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574560.260846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574561.220917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574561.284809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574562.244833:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574562.308832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574562.820846:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.820855:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574562.820859:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.820881:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6313 00000100:00080000:2.0:1788574562.820891:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.820898:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574562.820933:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864203904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574562.820954:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.820959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574562.820961:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.820966:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6313 00000100:00080000:2.0:1788574562.820970:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.820975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574562.820981:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864203904:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574562.820986:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574562.821005:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864204032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574562.821022:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.821025:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574562.821027:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.821030:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6313 00000100:00080000:2.0:1788574562.821034:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574562.821037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574562.821041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574562.821047:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574562.821059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864204160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574562.821066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574562.821068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574562.821069:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574562.821073:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6313 00000100:00080000:2.0:1788574562.821076:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574562.821079:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff92fc4617d880 x1875449864203904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574562.821082:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574562.821088:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574562.821099:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864204288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574562.821101:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864203904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574562.821121:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574562.821124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574562.821125:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574562.821129:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574562.821132:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788574562.821136:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6313 00000100:00100000:3.0:1788574562.821140:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864203904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574562.821141:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574562.821148:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574562.821152:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574562.821158:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864204416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574562.821165:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574562.821183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574562.821192:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff92fc4617e300 x1875449864204032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.821200:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864204032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.821206:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.821209:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574562.821219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574562.821224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574562.821236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574562.821243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff92fc4617d500 x1875449864204160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.821250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864204160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.821256:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.821258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574562.821260:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574562.821263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574562.821274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574562.821279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff92fc4617ea00 x1875449864204288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.821286:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864204288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.821291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.821293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574562.821295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574562.821306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574562.821316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574562.821323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff92fc4617ce00 x1875449864204416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574562.821330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864204416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574562.821336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574562.821338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574562.821340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574563.268887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574563.332836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574564.292931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574564.356847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574565.316875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574565.380917:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574566.340905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574566.404792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574567.364882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574567.428780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574567.940827:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.940861:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574567.940866:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.940878:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6318 00000100:00080000:2.0:1788574567.940890:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.940898:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574567.940948:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864204544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574567.940979:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.940990:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574567.940992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.940997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6318 00000100:00080000:2.0:1788574567.941001:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.941006:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574567.941019:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574567.941034:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864204672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574567.941045:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864204544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574567.941046:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.941054:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574567.941056:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.941060:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6318 00000100:00080000:2.0:1788574567.941063:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.941066:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574567.941070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574567.941082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864204800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574567.941089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.941092:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574567.941094:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.941096:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6318 00000100:00080000:2.0:1788574567.941112:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.941115:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574567.941120:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574567.941129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574567.941130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864204928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574567.941140:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574567.941143:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574567.941144:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574567.941147:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6318 00000100:00080000:2.0:1788574567.941149:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574567.941152:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574567.941154:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574567.941161:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864205056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574567.941192:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff92fc4617c380 x1875449864204544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.941215:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864204544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.941225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.941229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574567.941233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864204544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574567.941259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574567.941281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574567.941292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff92fc4617d180 x1875449864204672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.941301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864204672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.941312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.941315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574567.941317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574567.941328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574567.941342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574567.941351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff92fc4617e680 x1875449864204800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.941359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864204800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.941365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.941368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574567.941370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574567.941376:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574567.941403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574567.941413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff92fc4617c000 x1875449864204928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.941423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864204928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.941429:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.941431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574567.941434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864204928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574567.941439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574567.941459:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574567.941568:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff92fc4617fb80 x1875449864205056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.941711:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864205056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574567.941753:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.941807:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574567.941812:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574568.388953:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574568.452894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574569.412931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574569.476888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574570.436887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574570.500878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574571.460855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574571.524941:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574572.484939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574572.548861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574573.060917:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574573.060934:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574573.060940:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574573.060954:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6323 00000100:00080000:1.0:1788574573.060968:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574573.060978:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574573.061031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864205184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574573.061064:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574573.061067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574573.061068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574573.061072:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6323 00000100:00080000:1.0:1788574573.061074:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574573.061077:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574573.061087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574573.061097:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864205312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574573.061105:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574573.061108:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574573.061109:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574573.061112:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6323 00000100:00080000:1.0:1788574573.061113:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574573.061116:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574573.061119:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574573.061126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864205440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574573.061134:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864205184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574573.061135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574573.061141:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574573.061144:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574573.061149:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6323 00000100:00080000:1.0:1788574573.061152:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574573.061154:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574573.061157:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574573.061167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864205568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574573.061173:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574573.061175:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574573.061176:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574573.061178:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6323 00000100:00080000:1.0:1788574573.061180:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574573.061182:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574573.061185:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574573.061193:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864205696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574573.061241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574573.061296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574573/real 1788574573] req@ffff92fc6728aa00 x1875449864205184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574573.061318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864205184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574573.061331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574573.061336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574573.061340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864205184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574573.061349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574573.061374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574573.061388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574573/real 1788574573] req@ffff92fc6728b480 x1875449864205312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574573.061402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864205312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574573.061413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574573.061416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574573.061420:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574573.061450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574573.061473:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574573.061487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574573/real 1788574573] req@ffff92fc6728a300 x1875449864205440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574573.061500:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864205440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574573.061508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574573.061512:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574573.061516:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574573.061522:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574573.061544:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574573.061556:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574573/real 1788574573] req@ffff92fc67288e00 x1875449864205568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574573.061569:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864205568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574573.061579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574573.061583:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574573.061586:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574573.061592:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574573.061614:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574573.061625:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574573/real 1788574573] req@ffff92fc67288000 x1875449864205696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574573.061637:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864205696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574573.061647:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574573.061650:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574573.061654:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574573.508976:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574573.572894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574574.532897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574574.596784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574575.556923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574575.620908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574576.580926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574576.644907:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574577.604914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574577.668853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574578.180886:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574578.180897:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574578.180902:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574578.180912:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6328 00000100:00080000:3.0:1788574578.180923:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574578.180931:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574578.180970:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864205824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574578.181003:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574578.181007:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574578.181009:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574578.181013:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6328 00000100:00080000:3.0:1788574578.181017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574578.181021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574578.181030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574578.181044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864205952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574578.181055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574578.181059:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574578.181061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574578.181065:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6328 00000100:00080000:3.0:1788574578.181068:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574578.181072:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574578.181077:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574578.181086:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864205824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574578.181101:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864206080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574578.181116:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574578.181120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574578.181123:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574578.181127:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6328 00000100:00080000:3.0:1788574578.181132:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574578.181138:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574578.181147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574578.181162:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864206208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574578.181173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574578.181175:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574578.181181:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574578.181183:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574578.181190:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6328 00000100:00080000:3.0:1788574578.181197:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574578.181202:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574578.181209:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574578.181222:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864206336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574578.181244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff92fc67339180 x1875449864205824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574578.181265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864205824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574578.181281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574578.181284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574578.181288:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864205824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574578.181295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574578.181316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574578.181326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff92fc67339f80 x1875449864205952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574578.181334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864205952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574578.181340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574578.181342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574578.181345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864205952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574578.181374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574578.181395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574578.181408:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff92fc6733ad80 x1875449864206080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574578.181420:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864206080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574578.181430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574578.181433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574578.181437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574578.181444:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574578.181466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574578.181479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff92fc67339c00 x1875449864206208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574578.181492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864206208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574578.181501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574578.181505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574578.181509:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574578.181516:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574578.181534:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574578.181542:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574578/real 1788574578] req@ffff92fc67339880 x1875449864206336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574578.181550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864206336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574578.181556:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574578.181558:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574578.181560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574578.628943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574578.692895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574579.652910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574579.716917:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574580.676943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574580.740891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574581.700835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574581.764832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574582.724874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574582.788841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574583.300849:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574583.300860:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574583.300864:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574583.300874:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6333 00000100:00080000:2.0:1788574583.300885:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574583.300892:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574583.300950:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864206464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574583.300976:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574583.300981:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574583.300983:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574583.300987:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6333 00000100:00080000:2.0:1788574583.300992:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574583.300997:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574583.301007:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574583.301012:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864206464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574583.301024:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864206592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574583.301037:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574583.301040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574583.301042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574583.301046:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6333 00000100:00080000:2.0:1788574583.301051:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574583.301055:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574583.301060:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574583.301072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864206720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574583.301078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574583.301089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574583.301093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574583.301095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574583.301100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6333 00000100:00080000:2.0:1788574583.301104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574583.301108:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574583.301113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574583.301124:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff92fc4617dc00 x1875449864206464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574583.301126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864206848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574583.301154:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864206464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574583.301158:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574583.301167:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574583.301168:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574583.301173:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6333 00000100:00080000:2.0:1788574583.301178:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574583.301179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574583.301185:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788574583.301186:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574583.301192:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864206464:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574583.301194:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574583.301207:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864206976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574583.301214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574583.301237:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574583.301247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff92fc4617ca80 x1875449864206592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574583.301256:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864206592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574583.301262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574583.301264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574583.301267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574583.301271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574583.301283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574583.301289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff92fc4617c700 x1875449864206720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574583.301297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864206720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574583.301303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574583.301305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574583.301307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574583.301310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574583.301321:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574583.301326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff92fc4617f800 x1875449864206848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574583.301333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864206848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574583.301338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574583.301340:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574583.301342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574583.301352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574583.301363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574583.301369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574583/real 1788574583] req@ffff92fc4617f100 x1875449864206976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574583.301377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864206976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574583.301382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574583.301384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574583.301386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864206976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574583.748835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574583.812861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574584.772901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574584.836852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574585.796971:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574585.860850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574586.820943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574586.884877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574587.844914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574587.908816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574588.420912:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574588.420924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574588.420929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574588.420939:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6339 00000100:00080000:3.0:1788574588.420950:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574588.420957:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574588.420995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864207104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574588.421021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574588.421024:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574588.421026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574588.421029:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6339 00000100:00080000:3.0:1788574588.421032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574588.421035:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574588.421044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574588.421054:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864207232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574588.421062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574588.421065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574588.421067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574588.421069:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6339 00000100:00080000:3.0:1788574588.421071:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574588.421073:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574588.421076:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574588.421082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864207360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574588.421090:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574588.421093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574588.421095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574588.421101:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6339 00000100:00100000:0.0:1788574588.421103:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864207104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574588.421104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574588.421112:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574588.421124:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574588.421131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864207488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574588.421146:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574588.421148:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574588.421149:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574588.421153:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6339 00000100:00080000:3.0:1788574588.421155:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574588.421158:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574588.421162:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574588.421169:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864207616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574588.421217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574588.421274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff92fc6733b800 x1875449864207104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574588.421296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864207104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574588.421309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574588.421314:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574588.421319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864207104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574588.421329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574588.421355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574588.421368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff92fc67338a80 x1875449864207232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574588.421380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864207232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574588.421389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574588.421393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574588.421396:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574588.421428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574588.421448:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574588.421459:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff92fc67338380 x1875449864207360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574588.421470:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864207360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574588.421477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574588.421480:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574588.421483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574588.421489:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574588.421506:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574588.421517:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff92fc6733b480 x1875449864207488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574588.421530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864207488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574588.421538:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574588.421541:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574588.421545:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574588.421551:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574588.421569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574588.421579:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574588/real 1788574588] req@ffff92fc6733aa00 x1875449864207616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574588.421591:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864207616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574588.421599:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574588.421602:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574588.421605:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574588.868894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574588.932911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574589.892862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574589.956811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574590.916877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574590.980848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574591.940885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574592.004856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574592.964863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574593.028871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574593.540865:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574593.540880:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574593.540885:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574593.540898:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6344 00000100:00080000:3.0:1788574593.540912:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574593.540920:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574593.540968:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864207744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574593.541006:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574593.541010:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574593.541011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574593.541015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6344 00000100:00080000:3.0:1788574593.541017:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574593.541019:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574593.541031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574593.541043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864207872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574593.541051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574593.541054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574593.541055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574593.541058:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6344 00000100:00080000:3.0:1788574593.541060:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574593.541062:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574593.541066:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574593.541071:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864207744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574593.541077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864208000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574593.541085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574593.541087:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574593.541088:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574593.541090:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6344 00000100:00080000:3.0:1788574593.541093:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574593.541096:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574593.541099:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574593.541107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864208128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574593.541113:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574593.541115:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574593.541117:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574593.541120:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6344 00000100:00080000:3.0:1788574593.541124:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574593.541126:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574593.541129:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574593.541136:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864208256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574593.541185:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574593.541217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574593.541236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574593.541252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574593.988923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574594.052880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574595.012858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574595.076835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574596.036918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574596.100889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788574596.612984:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788574596.613032:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574596.613051:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574596.613105:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574596.613121:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574596.613135:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574596.613149:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788574596.613254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff92fc6733bb80 x1875449864208000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574596.613291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864208000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574596.613305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574596.613310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574596.613314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574596.613322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff92fc67338000 x1875449864208128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574596.613334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864208128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574596.613342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574596.613346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574596.613350:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574596.613357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff92fc67338700 x1875449864208256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574596.613376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864208256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574596.613385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574596.613389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574596.613393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208256:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574596.613400:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff92fc67339500 x1875449864207744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574596.613414:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864207744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574596.613422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574596.613425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574596.613428:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864207744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574596.613434:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574593/real 1788574596] req@ffff92fc6733b100 x1875449864207872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574596.613445:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864207872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574596.613455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574596.613459:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574596.613462:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864207872:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788574597.060846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574597.124831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574598.084890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574598.148849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574598.660836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574598.660847:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574598.660851:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574598.660862:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6349 00000100:00080000:0.0:1788574598.660872:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574598.660880:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574598.660914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864208384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574598.660941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574598.660945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574598.660947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574598.660951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6349 00000100:00080000:0.0:1788574598.660956:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574598.660960:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574598.660971:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574598.660980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864208384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574598.660988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864208512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574598.661002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574598.661005:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574598.661007:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574598.661012:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6349 00000100:00080000:0.0:1788574598.661016:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574598.661020:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574598.661025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574598.661038:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864208640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574598.661054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574598.661057:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574598.661059:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574598.661065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574598.661067:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6349 00000100:00080000:0.0:1788574598.661073:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574598.661078:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574598.661084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574598.661099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864208768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574598.661110:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574598.661114:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff92fc6704b100 x1875449864208384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574598.661116:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574598.661120:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574598.661128:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6349 00000100:00080000:0.0:1788574598.661135:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574598.661139:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574598.661144:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574598.661158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864208384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574598.661161:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864208896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574598.661188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574598.661193:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574598.661199:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864208384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574598.661231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574598.661271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574598.661289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff92fc67049880 x1875449864208512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574598.661302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864208512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574598.661316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574598.661320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574598.661323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574598.661329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574598.661363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574598.661376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff92fc6704ad80 x1875449864208640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574598.661389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864208640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574598.661397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574598.661400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574598.661404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574598.661411:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574598.661432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574598.661445:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff92fc6704a680 x1875449864208768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574598.661456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864208768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574598.661465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574598.661467:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574598.661470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574598.661476:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574598.661493:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574598.661503:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff92fc6704aa00 x1875449864208896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574598.661521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864208896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574598.661528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574598.661531:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574598.661535:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864208896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574599.108874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574599.172856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574600.132876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574600.196842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574601.156832:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574601.220820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574602.180926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574602.244914:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574603.204906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574603.268855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574603.780811:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.780820:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574603.780824:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.780833:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6354 00000100:00080000:0.0:1788574603.780842:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.780848:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574603.780879:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864209024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574603.780905:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.780909:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574603.780911:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.780915:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6354 00000100:00080000:0.0:1788574603.780918:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.780921:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574603.780930:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574603.780938:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864209024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574603.780943:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864209152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574603.780959:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.780963:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574603.780964:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.780968:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6354 00000100:00080000:0.0:1788574603.780971:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.780976:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574603.780981:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574603.780993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864209280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574603.781001:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.781004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574603.781007:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574603.781008:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.781015:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6354 00000100:00080000:0.0:1788574603.781031:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.781036:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574603.781041:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574603.781054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864209408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574603.781061:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff92fc6704a300 x1875449864209024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574603.781064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574603.781071:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574603.781074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574603.781083:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6354 00000100:00080000:0.0:1788574603.781086:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574603.781089:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574603.781092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574603.781098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864209536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574603.781185:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864209024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.781194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.781198:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574603.781200:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864209024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574603.781310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574603.781376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574603.781396:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff92fc67048000 x1875449864209152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.781412:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864209152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.781427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.781431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574603.781435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574603.781442:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574603.781464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574603.781476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff92fc67049f80 x1875449864209280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.781489:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864209280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.781497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.781499:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574603.781502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574603.781506:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574603.781522:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574603.781530:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff92fc67048700 x1875449864209408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.781538:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864209408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.781543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.781544:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574603.781547:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574603.781551:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574603.781561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574603.781567:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff92fc67048380 x1875449864209536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574603.781574:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864209536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574603.781579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574603.781581:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574603.781583:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574604.228865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574604.292844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574605.252873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574605.316802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574606.276804:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574606.340824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574607.300883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574607.364788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574608.324839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574608.388787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574608.900841:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.900853:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574608.900857:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.900869:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6359 00000100:00080000:2.0:1788574608.900879:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.900888:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574608.900929:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864209664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574608.900973:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.900979:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574608.900982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.900988:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6359 00000100:00080000:2.0:1788574608.900994:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.901000:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574608.901009:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864209664:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574608.901011:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574608.901032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864209792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574608.901054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.901059:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574608.901061:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574608.901065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574608.901068:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6359 00000100:00080000:2.0:1788574608.901076:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.901081:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574608.901088:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574608.901105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864209920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574608.901109:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff92fc4617ed80 x1875449864209664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574608.901139:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574608.901143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864209664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574608.901146:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574608.901151:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574608.901157:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574608.901159:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6359 00000100:00080000:3.0:1788574608.901164:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788574608.901169:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574608.901170:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864209664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574608.901180:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574608.901187:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574608.901201:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864210048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574608.901213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209792:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788574608.901215:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.901224:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574608.901226:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.901231:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6359 00000100:00080000:2.0:1788574608.901237:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.901242:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574608.901248:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574608.901251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574608.901265:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864210176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574608.901269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff92fc4617f480 x1875449864209792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.901294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864209792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.901302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.901305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574608.901308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574608.901314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574608.901329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574608.901336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff92fc4617f100 x1875449864209920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.901344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864209920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.901350:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.901353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574608.901355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864209920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574608.901364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574608.901375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574608.901382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff92fc4617f800 x1875449864210048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.901390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864210048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.901395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.901398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574608.901400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574608.901403:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574608.901413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574608.901419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574608] req@ffff92fc4617c700 x1875449864210176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574608.901426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864210176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574608.901431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574608.901433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574608.901435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574609.348868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574609.412785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574610.372865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574610.436783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574611.396864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574611.460803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574612.420824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574612.484787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574613.444809:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574613.508792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574614.020834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574614.020842:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574614.020845:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574614.020854:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6364 00000100:00080000:2.0:1788574614.020863:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574614.020871:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574614.020913:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864210304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574614.020943:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574614.020947:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574614.020949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574614.020954:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6364 00000100:00080000:2.0:1788574614.020958:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574614.020963:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574614.020973:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574614.020988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864210432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574614.020996:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574614.021001:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574614.021004:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574614.021007:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6364 00000100:00080000:2.0:1788574614.021013:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574614.021017:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864210304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574614.021018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574614.021032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574614.021041:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864210560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574614.021051:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574614.021054:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574614.021055:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574614.021058:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6364 00000100:00080000:2.0:1788574614.021061:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574614.021063:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574614.021067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574614.021076:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864210688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574614.021082:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574614.021084:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574614.021086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574614.021088:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6364 00000100:00080000:2.0:1788574614.021091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574614.021094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574614.021098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574614.021109:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574614.021110:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864210816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574614.021165:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574614/real 1788574614] req@ffff92fc4617ca80 x1875449864210304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574614.021182:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864210304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574614.021192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574614.021196:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574614.021198:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864210304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574614.021205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574614.021222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574614.021231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574614/real 1788574614] req@ffff92fc4617dc00 x1875449864210432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574614.021240:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864210432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574614.021246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574614.021248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574614.021251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574614.021278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574614.021290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574614.021297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574614/real 1788574614] req@ffff92fc4617fb80 x1875449864210560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574614.021305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864210560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574614.021311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574614.021313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574614.021315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574614.021319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574614.021330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574614.021337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574614/real 1788574614] req@ffff92fc4617c000 x1875449864210688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574614.021345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864210688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574614.021351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574614.021353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574614.021355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574614.021359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574614.021369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574614.021375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574614/real 1788574614] req@ffff92fc4617e680 x1875449864210816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574614.021382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864210816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574614.021388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574614.021389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574614.021391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864210816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574614.468858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574614.532820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574615.492862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574615.556863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574616.516828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574616.580838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574617.540890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574617.604836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574618.564813:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574618.628837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574619.140848:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574619.140856:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574619.140859:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574619.140867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6369 00000100:00080000:0.0:1788574619.140877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574619.140883:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574619.140913:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864210944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574619.140936:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574619.140939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574619.140940:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574619.140943:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6369 00000100:00080000:0.0:1788574619.140945:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574619.140947:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574619.140956:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574619.140966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864211072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574619.140974:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574619.140976:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574619.140978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574619.140980:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6369 00000100:00080000:0.0:1788574619.140983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574619.140985:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574619.140988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574619.140991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864210944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574619.140997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864211200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574619.141005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574619.141007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574619.141009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574619.141011:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6369 00000100:00080000:0.0:1788574619.141013:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574619.141015:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574619.141018:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574619.141025:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864211328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574619.141031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574619.141033:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574619.141034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574619.141037:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6369 00000100:00080000:0.0:1788574619.141040:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574619.141042:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574619.141045:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574619.141052:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864211456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574619.141064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574619.141111:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574619/real 1788574619] req@ffff92fc67048a80 x1875449864210944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574619.141128:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864210944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574619.141140:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574619.141143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574619.141146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864210944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574619.141165:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574619.141183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574619.141193:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574619/real 1788574619] req@ffff92fc67048380 x1875449864211072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574619.141201:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864211072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574619.141207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574619.141209:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574619.141211:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574619.141215:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574619.141226:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574619.141232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574619/real 1788574619] req@ffff92fc67048700 x1875449864211200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574619.141240:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864211200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574619.141245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574619.141247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574619.141249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574619.141253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574619.141264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574619.141283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574619/real 1788574619] req@ffff92fc67049f80 x1875449864211328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574619.141291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864211328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574619.141297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574619.141299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574619.141301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574619.141305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574619.141316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574619.141322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574619/real 1788574619] req@ffff92fc67048000 x1875449864211456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574619.141329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864211456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574619.141334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574619.141336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574619.141338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574619.588862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574619.652853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574620.612846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574620.676860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574621.636880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574621.700823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574622.660828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574622.724830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574623.684849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574623.748828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574624.260880:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.260891:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574624.260901:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.260912:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6374 00000100:00080000:2.0:1788574624.260921:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.260928:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574624.260961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864211584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574624.260985:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.260988:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574624.260990:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.260993:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6374 00000100:00080000:2.0:1788574624.260995:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.260997:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574624.261005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574624.261014:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864211712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574624.261022:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.261025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574624.261027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.261029:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6374 00000100:00080000:2.0:1788574624.261033:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.261038:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574624.261038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864211584:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574624.261045:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574624.261061:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864211840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574624.261069:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.261072:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574624.261073:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.261076:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6374 00000100:00080000:2.0:1788574624.261085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.261088:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574624.261091:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574624.261096:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574624.261102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864211968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574624.261109:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574624.261112:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574624.261114:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574624.261116:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6374 00000100:00080000:2.0:1788574624.261118:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574624.261125:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574624.261129:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574624.261134:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff92fc4617d180 x1875449864211584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574624.261138:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864212096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574624.261170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864211584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.261179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.261183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574624.261187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864211584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574624.261196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574624.261218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574624.261232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff92fc4617c380 x1875449864211712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.261245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864211712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.261254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.261257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574624.261261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574624.261283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574624.261297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574624.261305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff92fc4617ce00 x1875449864211840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.261313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864211840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.261323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.261330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574624.261333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574624.261336:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574624.261348:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574624.261354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff92fc4617ea00 x1875449864211968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.261362:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864211968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.261367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.261369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574624.261371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864211968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574624.261374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574624.261384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574624.261390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574624/real 1788574624] req@ffff92fc4617d500 x1875449864212096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574624.261396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864212096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574624.261401:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574624.261403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574624.261405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574624.708842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574624.772811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574625.732904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574625.796854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574626.756871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574626.820797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574627.780863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574627.844856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574628.804901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574628.868863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574629.380843:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.380854:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574629.380858:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.380869:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6379 00000100:00080000:2.0:1788574629.380891:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.380900:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574629.380936:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864212224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.380958:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.380961:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574629.380962:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.380966:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6379 00000100:00080000:2.0:1788574629.380969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.380971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574629.380979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574629.380987:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864212352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.380994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.380997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574629.380999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.381002:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6379 00000100:00080000:2.0:1788574629.381006:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.381009:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574629.381014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574629.381028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864212480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.381040:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574629.381044:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864212224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574629.381046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574629.381051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.381057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6379 00000100:00080000:2.0:1788574629.381060:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.381063:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574629.381068:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574629.381079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864212608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574629.381086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574629.381089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574629.381090:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574629.381092:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6379 00000100:00080000:2.0:1788574629.381095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574629.381097:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574629.381100:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574629.381107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574629.381110:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864212736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574629.381160:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff92fc4617e300 x1875449864212224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.381174:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864212224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.381183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.381187:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574629.381190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864212224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574629.381196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574629.381212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574629.381221:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff92fc4617d880 x1875449864212352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.381229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864212352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.381234:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.381237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574629.381239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574629.381257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574629.381281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574629.381293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff92fc4617d500 x1875449864212480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.381306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864212480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.381316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.381319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574629.381323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574629.381329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574629.381345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574629.381353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff92fc4617ea00 x1875449864212608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.381361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864212608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.381366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.381368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574629.381371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574629.381375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574629.381392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574629.381403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574629/real 1788574629] req@ffff92fc4617ce00 x1875449864212736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574629.381416:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864212736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574629.381450:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574629.381453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574629.381455:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574629.828850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574629.892821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574630.852859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574630.916789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574631.876903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574631.940868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574632.900904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574632.964841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574633.924848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574633.988834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574634.500886:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574634.500898:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574634.500903:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574634.500920:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6385 00000100:00080000:3.0:1788574634.500932:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574634.500940:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574634.500992:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864212864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574634.501031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574634.501036:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574634.501038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574634.501041:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6385 00000100:00080000:3.0:1788574634.501046:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574634.501050:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574634.501063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574634.501077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864212992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574634.501091:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864212864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574634.501099:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574634.501103:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574634.501105:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574634.501109:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6385 00000100:00080000:3.0:1788574634.501112:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574634.501116:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574634.501121:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574634.501130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864213120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574634.501138:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574634.501141:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574634.501143:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574634.501146:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6385 00000100:00080000:3.0:1788574634.501150:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574634.501152:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574634.501157:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574634.501163:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574634.501167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864213248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574634.501177:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574634.501181:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574634.501182:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574634.501188:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6385 00000100:00080000:3.0:1788574634.501190:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574634.501193:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574634.501197:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574634.501208:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864213376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574634.501224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff92fc6733a680 x1875449864212864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574634.501248:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864212864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574634.501258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574634.501262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574634.501266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864212864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574634.501286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574634.501304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574634.501312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff92fc67338e00 x1875449864212992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574634.501320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864212992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574634.501335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574634.501337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574634.501340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864212992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574634.501344:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574634.501356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574634.501364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff92fc6733a300 x1875449864213120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574634.501386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864213120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574634.501397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574634.501399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574634.501401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574634.501406:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574634.501421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574634.501428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff92fc6733b100 x1875449864213248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574634.501436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864213248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574634.501441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574634.501443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574634.501445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574634.501448:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574634.501466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574634.501473:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574634/real 1788574634] req@ffff92fc67339500 x1875449864213376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574634.501481:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864213376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574634.501486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574634.501488:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574634.501490:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574634.948893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574635.012906:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574635.972868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574636.036830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574636.996882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574637.060830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574638.020839:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574638.084809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574639.044993:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574639.108868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574639.620803:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574639.620813:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574639.620818:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574639.620829:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6390 00000100:00080000:3.0:1788574639.620840:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574639.620849:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574639.620898:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864213504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574639.620936:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574639.620940:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574639.620942:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574639.620946:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6390 00000100:00080000:3.0:1788574639.620950:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574639.620954:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574639.620964:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574639.620979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864213632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574639.620989:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574639.620992:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574639.620994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574639.621001:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6390 00000100:00080000:3.0:1788574639.621005:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574639.621006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864213504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574639.621012:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574639.621023:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574639.621039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864213760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574639.621046:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574639.621049:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574639.621050:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574639.621052:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6390 00000100:00080000:3.0:1788574639.621055:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574639.621058:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788574639.621111:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574639.621118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574639.621132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864213888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574639.621139:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574639.621142:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574639.621143:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574639.621147:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6390 00000100:00080000:3.0:1788574639.621150:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574639.621154:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574639.621158:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574639.621166:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864214016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574639.621181:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff92fc67338700 x1875449864213504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574639.621202:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864213504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574639.621213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574639.621218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574639.621221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864213504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574639.621254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574639.621275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574639.621285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff92fc67338000 x1875449864213632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574639.621293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864213632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574639.621300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574639.621302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574639.621305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574639.621309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574639.621340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574639.621349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff92fc6733bb80 x1875449864213760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574639.621357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864213760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574639.621363:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574639.621366:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574639.621368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574639.621372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574639.621385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574639.621394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff92fc6733aa00 x1875449864213888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574639.621406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864213888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574639.621415:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574639.621418:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574639.621422:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864213888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574639.621429:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574639.621453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574639.621462:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff92fc6733b480 x1875449864214016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574639.621475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864214016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574639.621481:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574639.621484:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574639.621486:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574640.068815:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574640.132809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574641.092876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574641.156824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574642.116882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574642.180801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574643.140957:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574643.204899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574644.164954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574644.228862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574644.740887:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574644.740902:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574644.740916:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574644.740928:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6395 00000100:00080000:1.0:1788574644.740940:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574644.740950:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574644.740998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864214144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574644.741026:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574644.741032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574644.741033:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574644.741036:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6395 00000100:00080000:1.0:1788574644.741039:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574644.741041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574644.741051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574644.741060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864214272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574644.741068:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574644.741071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574644.741072:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574644.741076:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6395 00000100:00080000:1.0:1788574644.741079:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574644.741082:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574644.741085:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574644.741091:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864214400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574644.741099:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574644.741101:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574644.741102:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574644.741107:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6395 00000100:00080000:1.0:1788574644.741111:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574644.741117:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574644.741119:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864214144:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788574644.741123:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574644.741137:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864214528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574644.741144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574644.741147:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574644.741148:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574644.741151:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6395 00000100:00080000:1.0:1788574644.741156:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574644.741158:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574644.741161:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574644.741169:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864214656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574644.741216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574644.741260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff92fc67289f80 x1875449864214144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574644.741314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864214144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574644.741326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574644.741330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574644.741334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864214144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574644.741343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574644.741368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574644.741381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff92fc67289880 x1875449864214272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574644.741393:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864214272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574644.741402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574644.741406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574644.741410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574644.741445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574644.741467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574644.741479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff92fc6728ad80 x1875449864214400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574644.741492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864214400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574644.741500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574644.741504:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574644.741508:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574644.741514:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574644.741533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574644.741545:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff92fc6728b800 x1875449864214528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574644.741557:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864214528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574644.741566:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574644.741569:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574644.741573:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574644.741589:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574644.741612:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574644.741625:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff92fc6728a680 x1875449864214656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574644.741638:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864214656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574644.741646:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574644.741650:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574644.741653:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574645.188898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574645.252868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574646.212939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574646.276784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574647.236871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574647.300827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574648.260903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574648.324793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574649.284839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574649.348848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574649.860831:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.860840:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574649.860844:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.860853:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6400 00000100:00080000:1.0:1788574649.860862:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.860868:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574649.860901:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864214784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.860924:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.860927:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574649.860929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.860931:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6400 00000100:00080000:1.0:1788574649.860934:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.860937:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574649.860944:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574649.860952:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864214912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.860959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.860962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574649.860964:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.860967:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6400 00000100:00080000:1.0:1788574649.860968:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.860971:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574649.860973:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574649.860980:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864215040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574649.860989:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864214784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574649.860989:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.860995:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574649.860998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.861002:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6400 00000100:00080000:1.0:1788574649.861004:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.861006:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574649.861009:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574649.861018:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864215168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.861029:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.861032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574649.861033:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.861036:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6400 00000100:00080000:1.0:1788574649.861046:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.861049:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574649.861051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574649.861056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574649.861060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864215296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574649.861097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff92fc67289c00 x1875449864214784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574649.861112:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864214784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574649.861120:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574649.861123:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574649.861125:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864214784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574649.861131:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574649.861145:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574649.861153:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff92fc67288380 x1875449864214912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574649.861161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864214912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574649.861166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574649.861169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574649.861171:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864214912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574649.861189:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574649.861199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574649.861206:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff92fc6728b100 x1875449864215040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574649.861214:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864215040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574649.861239:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574649.861241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574649.861244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574649.861248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574649.861263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574649.861270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff92fc67288a80 x1875449864215168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574649.861278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864215168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574649.861283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574649.861285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574649.861287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574649.861290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574649.861305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574649.861315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff92fc67288700 x1875449864215296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574649.861327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864215296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574649.861343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574649.861347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574649.861351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574650.308863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574650.372795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574651.332887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574651.396827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574652.356902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574652.420820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574653.380834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574653.444832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574654.404856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574654.468796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574654.980811:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574654.980820:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574654.980823:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574654.980832:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6405 00000100:00080000:2.0:1788574654.980841:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574654.980847:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574654.980880:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864215424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574654.980903:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574654.980906:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574654.980907:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574654.980910:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6405 00000100:00080000:2.0:1788574654.980912:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574654.980914:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574654.980922:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574654.980932:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864215552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574654.980941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574654.980943:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574654.980944:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574654.980949:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6405 00000100:00100000:0.0:1788574654.980951:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864215424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574654.980952:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574654.980958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574654.980965:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574654.980972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864215680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574654.980978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574654.980980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574654.980981:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574654.980983:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6405 00000100:00080000:2.0:1788574654.980985:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574654.980987:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574654.980990:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574654.980998:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864215808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574654.981003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574654.981005:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574654.981006:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574654.981008:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6405 00000100:00080000:2.0:1788574654.981010:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574654.981012:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574654.981014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574654.981021:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864215936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574654.981034:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574654.981069:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff92fc4617c380 x1875449864215424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574654.981083:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864215424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574654.981091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574654.981094:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574654.981097:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864215424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574654.981115:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574654.981130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574654.981139:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff92fc4617d180 x1875449864215552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574654.981147:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864215552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574654.981153:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574654.981155:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574654.981157:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574654.981160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574654.981197:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574654.981205:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff92fc4617e680 x1875449864215680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574654.981216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864215680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574654.981225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574654.981229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574654.981232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574654.981238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574654.981259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574654.981267:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff92fc4617c000 x1875449864215808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574654.981275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864215808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574654.981281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574654.981283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574654.981285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574654.981288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574654.981300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574654.981305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff92fc4617fb80 x1875449864215936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574654.981312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864215936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574654.981317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574654.981319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574654.981320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864215936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574655.428881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574655.492780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574656.452884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574656.516826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574657.476857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574657.540891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574658.500833:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574658.564843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574659.524850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574659.588888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574660.100860:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574660.100871:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574660.100875:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574660.100886:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6410 00000100:00080000:2.0:1788574660.100896:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574660.100904:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574660.100941:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864216064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574660.100977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574660.100981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574660.100984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574660.100988:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6410 00000100:00080000:2.0:1788574660.100992:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574660.100995:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574660.101005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574660.101019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864216192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574660.101026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864216064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574660.101037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574660.101042:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574660.101044:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574660.101049:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6410 00000100:00080000:2.0:1788574660.101053:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574660.101057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574660.101063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574660.101075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864216320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574660.101088:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574660.101091:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574660.101093:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574660.101097:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6410 00000100:00080000:2.0:1788574660.101102:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574660.101106:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574660.101127:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574660.101141:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864216448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574660.101147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216192:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788574660.101166:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574660.101171:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574660.101174:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574660.101179:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6410 00000100:00100000:0.0:1788574660.101184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216320:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788574660.101184:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574660.101193:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574660.101200:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574660.101220:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864216576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574660.101222:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574660.101250:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574660.548853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574660.612881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574661.572815:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574661.636862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574662.596858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574662.660833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788574663.172997:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:3.0:1788574663.173034:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788574663.173046:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788574663.173097:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788574663.173108:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788574663.173117:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788574663.173127:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788574663.173156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574660/real 1788574663] req@ffff92fc4617f100 x1875449864216576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574663.173184:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864216576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574663.173196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574663.173202:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574663.173207:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216576:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574663.173216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574660/real 1788574663] req@ffff92fc4617f800 x1875449864216448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574663.173228:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864216448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574663.173236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574663.173240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574663.173243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574663.173250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574660/real 1788574663] req@ffff92fc4617ca80 x1875449864216192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574663.173264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864216192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574663.173273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574663.173277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574663.173281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574663.173288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574660/real 1788574663] req@ffff92fc4617c700 x1875449864216320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574663.173302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864216320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574663.173309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574663.173313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574663.173316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574663.173323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574660/real 1788574663] req@ffff92fc4617dc00 x1875449864216064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574663.173335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864216064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574663.173344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574663.173348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574663.173353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864216064:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788574663.620852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574663.684814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574664.644936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574664.708897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574665.220843:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574665.220855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574665.220859:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574665.220870:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6415 00000100:00080000:2.0:1788574665.220881:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574665.220889:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574665.220923:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864216704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574665.220954:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574665.220959:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574665.220961:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574665.220965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6415 00000100:00080000:2.0:1788574665.220969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574665.220973:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574665.220983:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574665.221001:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864216832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574665.221012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574665.221016:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574665.221017:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574665.221022:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6415 00000100:00080000:2.0:1788574665.221027:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574665.221032:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574665.221038:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574665.221046:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864216704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574665.221094:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864216960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574665.221112:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574665.221117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574665.221119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574665.221124:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6415 00000100:00080000:2.0:1788574665.221129:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574665.221134:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574665.221140:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574665.221153:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864217088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574665.221164:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574665.221171:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788574665.221173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574665.221176:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574665.221185:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6415 00000100:00080000:2.0:1788574665.221190:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574665.221195:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574665.221206:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574665.221226:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864217216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574665.221265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff92fc4617f480 x1875449864216704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574665.221299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864216704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574665.221312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574665.221317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574665.221322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864216704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574665.221331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574665.221372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574665.221388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff92fc4617ed80 x1875449864216832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574665.221408:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864216832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574665.221421:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574665.221425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574665.221430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574665.221468:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574665.221491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574665.221504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff92fc4617dc00 x1875449864216960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574665.221518:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864216960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574665.221527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574665.221531:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574665.221535:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864216960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574665.221543:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574665.221567:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574665.221588:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff92fc4617c700 x1875449864217088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574665.221602:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864217088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574665.221611:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574665.221615:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574665.221618:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574665.221632:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574665.221658:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574665.221672:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574665/real 1788574665] req@ffff92fc4617ca80 x1875449864217216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574665.221684:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864217216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574665.221693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574665.221696:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574665.221700:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574665.668879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574665.732805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574666.692861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574666.756830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574667.716849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574667.780844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574668.740855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574668.804830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574669.764826:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574669.828778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574670.340847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.340856:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574670.340859:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.340868:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6420 00000100:00080000:3.0:1788574670.340876:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.340882:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574670.340919:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864217344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.340962:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.340967:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574670.340969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.340974:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6420 00000100:00080000:3.0:1788574670.340983:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.340989:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574670.341000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574670.341022:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864217344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574670.341031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864217472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.341046:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.341051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574670.341053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.341057:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6420 00000100:00080000:3.0:1788574670.341062:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.341066:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574670.341072:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574670.341086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574670.341088:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864217600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.341101:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574670.341104:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574670.341106:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.341109:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6420 00000100:00080000:3.0:1788574670.341113:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.341117:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574670.341122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574670.341124:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff92fc67338380 x1875449864217344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574670.341133:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864217728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574670.341141:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574670.341145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864217344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574670.341146:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574670.341149:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574670.341156:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6420 00000100:00080000:3.0:1788574670.341161:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574670.341166:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788574670.341167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574670.341172:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574670.341174:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574670.341179:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864217344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574670.341183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864217856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574670.341204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574670.341220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574670.341228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff92fc67338a80 x1875449864217472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574670.341236:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864217472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574670.341242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574670.341244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574670.341246:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574670.341250:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574670.341260:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574670.341266:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff92fc6733b800 x1875449864217600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574670.341273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864217600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574670.341278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574670.341280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574670.341282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574670.341285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574670.341295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574670.341301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff92fc67339880 x1875449864217728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574670.341309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864217728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574670.341314:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574670.341316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574670.341318:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574670.341321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574670.341330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574670.341335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574670/real 1788574670] req@ffff92fc67339c00 x1875449864217856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574670.341342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864217856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574670.341347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574670.341349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574670.341351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864217856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574670.788880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574670.852836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574671.812928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574671.876874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574672.836841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574672.900804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574673.860840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574673.924834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574674.884889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574674.948829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574675.460830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574675.460842:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574675.460847:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574675.460858:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6426 00000100:00080000:3.0:1788574675.460868:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574675.460876:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574675.460912:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864217984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574675.460973:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574675.460978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574675.460980:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574675.460985:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6426 00000100:00080000:3.0:1788574675.460989:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574675.460995:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574675.461005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574675.461022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864218112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574675.461032:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574675.461035:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574675.461038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574675.461042:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6426 00000100:00080000:3.0:1788574675.461047:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574675.461051:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574675.461056:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574675.461069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864218240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574675.461079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574675.461083:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574675.461084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574675.461088:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6426 00000100:00080000:3.0:1788574675.461091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574675.461095:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574675.461099:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574675.461110:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864218368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574675.461184:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574675.461188:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574675.461190:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574675.461195:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6426 00000100:00080000:3.0:1788574675.461199:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574675.461204:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574675.461210:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574675.461225:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864218496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574675.461312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864217984:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788574675.461359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574675.461387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff92fc6733ad80 x1875449864217984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574675.461404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864217984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574675.461415:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574675.461419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574675.461423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864217984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574675.461431:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574675.461464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574675.461479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff92fc67339f80 x1875449864218112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574675.461491:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864218112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574675.461500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574675.461503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574675.461507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574675.461513:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574675.461535:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574675.461547:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff92fc67339180 x1875449864218240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574675.461559:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864218240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574675.461569:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574675.461583:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574675.461586:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574675.461596:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574675.461619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574675.461633:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff92fc67339c00 x1875449864218368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574675.461644:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864218368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574675.461651:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574675.461654:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574675.461657:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574675.461663:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574675.461683:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574675.461696:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574675/real 1788574675] req@ffff92fc67339880 x1875449864218496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574675.461708:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864218496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574675.461716:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574675.461719:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574675.461746:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574675.908868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574675.972827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574676.932802:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574676.996809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574677.956879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574678.020823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574678.980825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574679.044811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574680.004920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574680.068829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574680.580812:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574680.580822:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574680.580827:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574680.580837:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6431 00000100:00080000:3.0:1788574680.580847:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574680.580853:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574680.580889:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864218624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574680.580948:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574680.580953:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574680.580955:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574680.580959:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6431 00000100:00080000:3.0:1788574680.580963:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574680.580967:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574680.580984:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574680.581000:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864218624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574680.581005:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864218752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574680.581020:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574680.581024:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574680.581026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574680.581031:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6431 00000100:00080000:3.0:1788574680.581035:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574680.581039:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574680.581046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574680.581060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864218880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574680.581066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574680.581077:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574680.581081:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574680.581083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574680.581089:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6431 00000100:00080000:3.0:1788574680.581093:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574680.581097:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574680.581104:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574680.581110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff92fc6733b800 x1875449864218624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574680.581119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864219008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574680.581135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574680.581140:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574680.581141:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864218624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574680.581143:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574680.581153:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6431 00000100:00080000:3.0:1788574680.581159:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.581163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574680.581166:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788574680.581171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:3.0:1788574680.581174:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574680.581178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864218624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574680.581188:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864219136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574680.581206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574680.581229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574680.581241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff92fc67338a80 x1875449864218752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574680.581254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864218752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574680.581263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574680.581266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574680.581270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574680.581276:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574680.581295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574680.581307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff92fc67338380 x1875449864218880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574680.581320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864218880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574680.581329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574680.581332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574680.581336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864218880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574680.581342:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574680.581359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574680.581369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff92fc6733b480 x1875449864219008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574680.581382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864219008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574680.581391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574680.581395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574680.581399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574680.581405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574680.581421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574680.581431:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff92fc6733aa00 x1875449864219136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574680.581443:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864219136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574680.581451:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574680.581454:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574680.581458:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574681.028887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574681.092824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574682.052866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574682.116838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574683.076961:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574683.140804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574684.100934:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574684.164920:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574685.125001:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574685.188844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574685.700844:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.700851:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574685.700855:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.700863:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6436 00000100:00080000:2.0:1788574685.700872:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.700877:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574685.700908:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864219264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.700931:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.700933:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574685.700935:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.700937:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6436 00000100:00080000:2.0:1788574685.700939:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.700941:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574685.700950:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574685.700958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864219392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.700964:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.700966:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574685.700967:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.700971:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6436 00000100:00080000:2.0:1788574685.700972:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.700975:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574685.700977:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574685.700983:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864219520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.700990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.700993:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574685.700994:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.700997:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6436 00000100:00080000:2.0:1788574685.700999:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.701002:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574685.701006:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574685.701013:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864219648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.701020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.701022:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574685.701024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574685.701031:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864219264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574685.701032:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6436 00000100:00080000:2.0:1788574685.701042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.701051:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574685.701055:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574685.701062:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864219776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574685.701129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574685.701181:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff92fc4617f800 x1875449864219264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.701212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864219264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.701223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.701227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574685.701231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864219264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574685.701236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574685.701254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574685.701263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff92fc4617f100 x1875449864219392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.701271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864219392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.701277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.701280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574685.701282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574685.701286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574685.701297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574685.701303:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff92fc4617fb80 x1875449864219520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.701315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864219520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.701321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.701322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574685.701324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574685.701352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574685.701365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574685.701372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff92fc4617c000 x1875449864219648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.701379:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864219648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.701385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.701387:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574685.701389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574685.701393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574685.701403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574685.701409:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff92fc4617e680 x1875449864219776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.701416:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864219776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.701433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.701436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574685.701438:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864219776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574686.148859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574686.212821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574687.172891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574687.236786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574688.196816:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574688.260825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574689.220897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574689.284773:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574690.244918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574690.308900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574690.820832:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.820842:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574690.820846:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.820857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6441 00000100:00080000:2.0:1788574690.820869:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.820878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574690.820917:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864219904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574690.820942:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.820948:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574690.820949:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.820952:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6441 00000100:00080000:2.0:1788574690.820955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.820958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574690.820966:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574690.820976:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864220032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574690.820983:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.820985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574690.820986:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.820989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6441 00000100:00080000:2.0:1788574690.820991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.820993:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574690.820996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574690.821003:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864220160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574690.821009:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864219904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574690.821012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.821018:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574690.821021:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.821024:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6441 00000100:00080000:2.0:1788574690.821026:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.821028:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574690.821037:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574690.821045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864220288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574690.821051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574690.821054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574690.821055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574690.821057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6441 00000100:00080000:2.0:1788574690.821059:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574690.821061:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574690.821064:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574690.821072:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864220416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574690.821079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574690.821119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff92fc4617d180 x1875449864219904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.821133:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864219904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.821141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.821143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574690.821146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864219904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574690.821151:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574690.821167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574690.821175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff92fc4617c380 x1875449864220032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.821183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864220032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.821189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.821191:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574690.821193:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574690.821213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574690.821259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574690.821268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff92fc4617ce00 x1875449864220160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.821275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864220160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.821281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.821283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574690.821285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574690.821290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574690.821300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574690.821306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff92fc4617ea00 x1875449864220288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.821314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864220288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.821319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.821321:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574690.821324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574690.821327:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574690.821336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574690.821342:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff92fc4617d500 x1875449864220416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.821349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864220416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.821354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.821356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574690.821358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574691.268968:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574691.332899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574692.292855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574692.356886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574693.316890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574693.380881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574694.340844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574694.404861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574695.364929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574695.428915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574695.940853:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.940868:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574695.940874:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.940888:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6446 00000100:00080000:0.0:1788574695.940902:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.940913:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574695.940974:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864220544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.941012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.941017:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574695.941022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.941028:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6446 00000100:00080000:0.0:1788574695.941032:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.941036:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574695.941046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574695.941059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864220672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574695.941063:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864220544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574695.941070:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.941077:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574695.941078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.941082:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6446 00000100:00080000:0.0:1788574695.941084:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.941086:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574695.941090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574695.941101:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864220800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.941114:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.941117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574695.941119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.941122:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6446 00000100:00080000:0.0:1788574695.941124:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.941128:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574695.941131:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574695.941137:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864220928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.941143:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.941146:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574695.941148:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.941150:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6446 00000100:00080000:0.0:1788574695.941154:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574695.941160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574695.941161:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574695.941166:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574695.941179:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864221056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574695.941212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff92fc6704a300 x1875449864220544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.941228:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864220544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.941237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.941240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574695.941243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864220544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574695.941266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574695.941284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574695.941302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff92fc6704aa00 x1875449864220672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.941311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864220672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.941317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.941320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574695.941322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574695.941327:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574695.941345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574695.941352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff92fc6704a680 x1875449864220800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.941359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864220800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.941365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.941367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574695.941369:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574695.941373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574695.941384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574695.941390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff92fc6704ad80 x1875449864220928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.941402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864220928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.941407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.941409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574695.941411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864220928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574695.941414:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574695.941425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574695.941430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff92fc67049880 x1875449864221056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.941437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864221056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.941450:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.941452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574695.941454:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574696.388863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574696.452850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574697.412913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574697.476800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574698.436929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574698.500887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574699.460907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574699.524865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574700.484865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574700.548821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574701.060898:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574701.060935:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574701.060958:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574701.060972:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6451 00000100:00080000:0.0:1788574701.060987:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574701.060997:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574701.061066:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864221184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574701.061111:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574701.061119:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574701.061122:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574701.061129:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6451 00000100:00080000:0.0:1788574701.061136:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574701.061144:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574701.061176:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574701.061198:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864221312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574701.061217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864221184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574701.061241:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574701.061249:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574701.061254:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574701.061266:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6451 00000100:00080000:0.0:1788574701.061274:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574701.061285:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574701.061333:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574701.061357:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864221440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574701.061363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574701.061389:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574701.061395:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574701.061398:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574701.061407:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6451 00000100:00080000:0.0:1788574701.061414:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574701.061423:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574701.061430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574701/real 1788574701] req@ffff92fc6704b100 x1875449864221184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574701.061436:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574701.061455:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864221184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574701.061467:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864221568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574701.061509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574701.061518:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788574701.061519:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574701.061524:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864221184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574701.061533:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574701.061539:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574701.061549:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6451 00000100:00080000:0.0:1788574701.061555:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574701.061565:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221312:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788574701.061568:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574701.061581:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574701.061596:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574701.061610:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864221696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574701.061615:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574701/real 1788574701] req@ffff92fc6704bb80 x1875449864221312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574701.061671:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864221312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574701.061683:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574701.061689:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574701.061696:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574701.061719:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574701.061794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574701.061803:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574701/real 1788574701] req@ffff92fc67049500 x1875449864221440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574701.061812:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864221440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574701.061818:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574701.061821:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574701.061823:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574701.061829:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574701.061842:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574701.061849:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574701/real 1788574701] req@ffff92fc67049180 x1875449864221568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574701.061857:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864221568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574701.061862:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574701.061864:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574701.061867:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574701.061880:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574701.061892:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574701.061898:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574701/real 1788574701] req@ffff92fc67049c00 x1875449864221696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574701.061905:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864221696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574701.061910:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574701.061913:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574701.061915:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574701.508815:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574701.572843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574702.532897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574702.596830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574703.556943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574703.620848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574704.580871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574704.644813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574705.604824:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574705.668830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574706.180875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574706.180884:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574706.180888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574706.180896:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6456 00000100:00080000:3.0:1788574706.180912:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574706.180918:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574706.180952:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864221824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574706.180981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574706.180986:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574706.180988:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574706.180993:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6456 00000100:00080000:3.0:1788574706.180998:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574706.181002:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574706.181012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574706.181029:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864221824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574706.181030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864221952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574706.181045:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574706.181048:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574706.181050:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574706.181053:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6456 00000100:00080000:3.0:1788574706.181056:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574706.181059:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574706.181063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574706.181072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864222080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574706.181083:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574706.181086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574706.181087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574706.181090:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6456 00000800:00000100:2.0:1788574706.181094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574706.181095:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574706.181100:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574706.181104:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574706.181112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864222208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574706.181118:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574706.181120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574706.181121:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574706.181124:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6456 00000100:00080000:3.0:1788574706.181127:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574706.181130:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574706.181133:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574706.181136:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff92fc6733bb80 x1875449864221824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574706.181144:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864222336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574706.181159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864221824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574706.181169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574706.181173:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574706.181177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864221824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574706.181195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574706.181210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574706.181219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff92fc67338000 x1875449864221952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574706.181227:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864221952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574706.181233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574706.181236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574706.181240:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864221952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574706.181246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574706.181266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574706.181277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff92fc67338700 x1875449864222080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574706.181288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864222080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574706.181302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574706.181305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574706.181308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574706.181314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574706.181345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574706.181356:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff92fc67339500 x1875449864222208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574706.181367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864222208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574706.181375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574706.181378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574706.181381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574706.181387:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574706.181406:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574706.181418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574706/real 1788574706] req@ffff92fc6733b100 x1875449864222336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574706.181430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864222336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574706.181436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574706.181440:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574706.181444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574706.628850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574706.692778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574707.652893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574707.716888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574708.676867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574708.740780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574709.700841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574709.764844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574710.724914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574710.788857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574711.300829:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574711.300845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574711.300851:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574711.300865:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6461 00000100:00080000:3.0:1788574711.300880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574711.300889:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574711.300943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864222464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574711.300983:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574711.300988:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574711.300991:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574711.300998:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6461 00000100:00080000:3.0:1788574711.301003:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574711.301008:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574711.301020:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574711.301033:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864222592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574711.301038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864222464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574711.301046:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574711.301050:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574711.301052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574711.301056:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6461 00000100:00080000:3.0:1788574711.301060:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574711.301063:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574711.301067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574711.301077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864222720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574711.301084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574711.301087:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574711.301089:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574711.301091:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6461 00000100:00080000:3.0:1788574711.301094:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574711.301096:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574711.301100:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574711.301118:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574711.301172:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff92fc6733a300 x1875449864222464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574711.301196:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864222848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574711.301197:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864222464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574711.301231:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574711.301232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574711.301235:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574711.301239:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574711.301241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788574711.301246:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6461 00000100:00100000:0.0:1788574711.301252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864222464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574711.301253:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574711.301263:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574711.301267:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574711.301276:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864222976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574711.301283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574711.301307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574711.301321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff92fc67338e00 x1875449864222592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574711.301337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864222592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574711.301347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574711.301349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574711.301352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574711.301357:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574711.301400:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574711.301410:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff92fc6733a680 x1875449864222720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574711.301419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864222720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574711.301426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574711.301428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574711.301430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574711.301434:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574711.301449:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574711.301456:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff92fc6733b100 x1875449864222848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574711.301465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864222848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574711.301477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574711.301481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574711.301484:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574711.301499:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574711.301513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574711.301519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574711/real 1788574711] req@ffff92fc67339500 x1875449864222976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574711.301528:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864222976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574711.301534:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574711.301536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574711.301539:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864222976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574711.748871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574711.812819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574712.772882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574712.836828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574713.796823:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574713.860831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574714.820865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574714.884822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574715.844817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574715.908810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574716.420828:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574716.420837:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574716.420840:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574716.420849:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6467 00000100:00080000:1.0:1788574716.420858:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574716.420864:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574716.420897:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864223104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574716.420989:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574716.420992:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574716.420993:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574716.420996:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6467 00000100:00080000:1.0:1788574716.420999:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574716.421001:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574716.421011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574716.421021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864223232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574716.421029:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574716.421032:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574716.421034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574716.421036:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864223104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574716.421038:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6467 00000100:00080000:1.0:1788574716.421043:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574716.421047:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574716.421050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574716.421058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864223360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574716.421064:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574716.421066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574716.421068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574716.421070:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6467 00000100:00080000:1.0:1788574716.421073:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574716.421075:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574716.421079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574716.421085:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864223488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574716.421091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574716.421093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574716.421095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574716.421100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574716.421103:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6467 00000100:00080000:1.0:1788574716.421107:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574716.421109:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574716.421112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574716.421120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864223616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574716.421149:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff92fc67289180 x1875449864223104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574716.421170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864223104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574716.421181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574716.421184:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574716.421188:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864223104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574716.421213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574716.421238:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574716.421252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff92fc67289500 x1875449864223232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574716.421261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864223232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574716.421268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574716.421270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574716.421272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574716.421277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574716.421291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574716.421298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff92fc6728bb80 x1875449864223360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574716.421316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864223360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574716.421321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574716.421323:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574716.421325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574716.421329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574716.421342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574716.421348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff92fc67288700 x1875449864223488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574716.421356:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864223488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574716.421361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574716.421363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574716.421365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574716.421368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574716.421378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574716.421384:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574716/real 1788574716] req@ffff92fc67288a80 x1875449864223616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574716.421396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864223616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574716.421401:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574716.421402:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574716.421404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574716.868894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574716.932817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574717.892845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574717.956805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574718.916927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574718.980905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574719.940891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574720.004832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574720.964857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574721.028831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574721.540847:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574721.540856:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574721.540859:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574721.540867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6472 00000100:00080000:2.0:1788574721.540876:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574721.540882:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574721.540913:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864223744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574721.540934:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574721.540938:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574721.540940:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574721.540944:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6472 00000100:00080000:2.0:1788574721.540948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574721.540953:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574721.540964:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574721.540976:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864223744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574721.540981:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864223872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574721.540995:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574721.540998:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574721.540999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574721.541002:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6472 00000100:00080000:2.0:1788574721.541005:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574721.541009:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574721.541013:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574721.541024:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864224000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574721.541031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574721.541034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574721.541035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574721.541037:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6472 00000100:00080000:2.0:1788574721.541040:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574721.541044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788574721.541045:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:2.0:1788574721.541049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574721.541058:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864224128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574721.541066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574721.541068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574721.541069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574721.541071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6472 00000100:00080000:2.0:1788574721.541074:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574721.541076:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574721.541079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574721.541091:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff92fc4617d880 x1875449864223744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574721.541092:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864224256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574721.541128:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864223744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574721.541136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574721.541139:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574721.541142:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864223744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574721.541162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574721.541179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574721.541198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff92fc4617e300 x1875449864223872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574721.541212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864223872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574721.541221:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574721.541225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574721.541229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864223872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574721.541236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574721.541253:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574721.541262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff92fc4617d500 x1875449864224000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574721.541270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864224000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574721.541275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574721.541277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574721.541279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574721.541283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574721.541295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574721.541301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff92fc4617ea00 x1875449864224128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574721.541308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864224128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574721.541313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574721.541315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574721.541317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574721.541321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574721.541331:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574721.541337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574721/real 1788574721] req@ffff92fc4617ce00 x1875449864224256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574721.541344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864224256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574721.541349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574721.541351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574721.541353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574721.988890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574722.052822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574723.012889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574723.076830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574724.036914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574724.100803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574725.060861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574725.124848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574726.084880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574726.148794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574726.660871:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574726.660882:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574726.660888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574726.660900:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6477 00000100:00080000:1.0:1788574726.660914:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574726.660923:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574726.660969:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864224384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574726.661005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574726.661009:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574726.661010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574726.661014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6477 00000100:00080000:1.0:1788574726.661017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574726.661020:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574726.661030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574726.661042:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864224512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574726.661055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574726.661058:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574726.661060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574726.661066:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6477 00000100:00100000:0.0:1788574726.661070:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864224384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574726.661071:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574726.661080:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574726.661089:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574726.661103:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864224640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574726.661109:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574726.661112:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574726.661114:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574726.661117:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6477 00000100:00080000:1.0:1788574726.661120:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574726.661122:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574726.661127:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574726.661143:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864224768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574726.661150:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574726.661152:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574726.661153:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574726.661156:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6477 00000100:00080000:1.0:1788574726.661161:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574726.661165:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574726.661169:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574726.661179:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864224896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574726.661195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574726.661225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574726.661242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574726.661256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574727.108858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574727.172849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574728.132880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574728.196779:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574729.156860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574729.220843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574729.733053:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574729.733104:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574729.733125:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574729.733181:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574729.733195:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574729.733207:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574729.733219:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788574729.733238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff92fc67288380 x1875449864224512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574729.733275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864224512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574729.733300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574729.733308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574729.733314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574729.733330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff92fc67289c00 x1875449864224640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574729.733362:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864224640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574729.733374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574729.733379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574729.733383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574729.733392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff92fc6728a680 x1875449864224768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574729.733407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864224768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574729.733416:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574729.733421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574729.733425:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574729.733435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff92fc6728b100 x1875449864224384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574729.733450:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864224384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574729.733459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574729.733463:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574729.733467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864224384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574729.733505:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574729] req@ffff92fc6728b800 x1875449864224896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574729.733521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864224896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574729.733531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574729.733535:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574729.733539:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864224896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574730.180874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574730.244844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574731.204805:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574731.268809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574731.780841:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574731.780849:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574731.780853:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574731.780861:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6482 00000100:00080000:3.0:1788574731.780869:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574731.780875:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574731.780903:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864225024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574731.780934:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574731.780936:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574731.780938:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574731.780941:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6482 00000100:00080000:3.0:1788574731.780944:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574731.780946:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574731.780954:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574731.780964:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864225152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574731.780971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574731.780974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574731.780975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574731.780979:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6482 00000100:00080000:3.0:1788574731.780983:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574731.780984:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864225024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574731.780990:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574731.781007:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574731.781023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864225280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574731.781034:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574731.781038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574731.781040:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574731.781044:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6482 00000100:00080000:3.0:1788574731.781058:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574731.781063:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788574731.781070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574731.781073:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574731.781089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864225408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574731.781097:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574731.781099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574731.781101:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574731.781104:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6482 00000100:00080000:3.0:1788574731.781107:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574731.781110:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574731.781113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574731.781126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864225536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574731.781128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff92fc67338700 x1875449864225024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.781163:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864225024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.781177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.781182:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574731.781187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864225024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574731.781196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574731.781223:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574731.781240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff92fc67338000 x1875449864225152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.781252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864225152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.781267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.781271:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574731.781274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574731.781303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788574731.781326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574731.781338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff92fc6733bb80 x1875449864225280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.781349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864225280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.781358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.781362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574731.781365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788574731.781371:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574731.781392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574731.781403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff92fc6733aa00 x1875449864225408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.781417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864225408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.781426:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.781429:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574731.781433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788574731.781438:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788574731.781460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574731.781471:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff92fc6733b480 x1875449864225536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.781482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864225536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.781491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.781494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788574731.781497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574732.228858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574732.292804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574733.252828:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574733.316826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574734.276838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574734.340831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574735.300801:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574735.364842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574736.324891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574736.388888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574736.900791:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574736.900809:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574736.900817:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574736.900827:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6487 00000100:00080000:0.0:1788574736.900837:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574736.900845:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574736.900883:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864225664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574736.900912:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574736.900916:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574736.900917:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574736.900920:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6487 00000100:00080000:0.0:1788574736.900923:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574736.900926:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574736.900934:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574736.900947:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864225792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574736.900954:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574736.900957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574736.900960:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574736.900961:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864225664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574736.900965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6487 00000100:00080000:0.0:1788574736.900971:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574736.900973:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574736.900976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574736.900985:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864225920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574736.900991:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574736.900994:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574736.900996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574736.900999:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6487 00000100:00080000:0.0:1788574736.901002:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574736.901004:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574736.901008:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574736.901016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864226048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574736.901023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574736.901027:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574736.901030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574736.901032:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574736.901036:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6487 00000100:00080000:0.0:1788574736.901042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574736.901044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574736.901061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574736.901075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864226176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574736.901078:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff92fc67048e00 x1875449864225664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574736.901113:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864225664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.901123:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574736.901127:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574736.901130:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864225664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574736.901156:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574736.901173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574736.901188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff92fc67049c00 x1875449864225792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574736.901197:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864225792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.901203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574736.901205:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574736.901209:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574736.901213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574736.901225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574736.901232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff92fc67049180 x1875449864225920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574736.901239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864225920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.901245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574736.901247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574736.901249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864225920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574736.901252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574736.901263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574736.901268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff92fc67049500 x1875449864226048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574736.901275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864226048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.901280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574736.901282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574736.901285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574736.901288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574736.901298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574736.901304:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff92fc6704bb80 x1875449864226176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574736.901311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864226176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.901316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574736.901318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574736.901320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574737.348863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574737.412848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574738.372972:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574738.436876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574739.396971:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574739.460804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574740.420894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574740.484797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574741.444911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574741.508843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574742.020845:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574742.020853:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574742.020857:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574742.020866:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6492 00000100:00080000:3.0:1788574742.020875:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574742.020888:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574742.020920:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864226304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574742.020942:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574742.020947:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574742.020949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574742.020951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6492 00000100:00080000:3.0:1788574742.020954:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574742.020956:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574742.020964:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574742.020974:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864226432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574742.020983:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574742.020991:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574742.020994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864226304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574742.020994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574742.021000:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6492 00000100:00080000:3.0:1788574742.021005:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574742.021008:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574742.021011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574742.021022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864226560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574742.021028:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574742.021031:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574742.021033:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574742.021035:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6492 00000100:00080000:3.0:1788574742.021037:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574742.021039:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574742.021043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574742.021055:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864226688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574742.021063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574742.021067:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574742.021068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574742.021072:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574742.021075:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6492 00000100:00080000:3.0:1788574742.021078:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574742.021080:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574742.021084:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574742.021092:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864226816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574742.021110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574742/real 1788574742] req@ffff92fc67338380 x1875449864226304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574742.021124:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864226304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574742.021132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574742.021136:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574742.021138:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864226304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574742.021157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574742.021172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574742.021181:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574742/real 1788574742] req@ffff92fc67338a80 x1875449864226432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574742.021189:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864226432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574742.021199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574742.021201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574742.021204:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574742.021207:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574742.021218:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574742.021224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574742/real 1788574742] req@ffff92fc6733b800 x1875449864226560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574742.021231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864226560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574742.021237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574742.021240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574742.021243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574742.021249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574742.021269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574742.021281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574742/real 1788574742] req@ffff92fc67339880 x1875449864226688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574742.021294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864226688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574742.021300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574742.021302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574742.021304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574742.021308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574742.021323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574742.021340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574742/real 1788574742] req@ffff92fc67339c00 x1875449864226816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574742.021347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864226816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574742.021353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574742.021355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574742.021357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864226816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574742.468927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574742.532844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574743.492863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574743.556801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574744.516859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574744.580848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574745.540826:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574745.604797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574746.564846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574746.628807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574747.140865:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574747.140877:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574747.140884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574747.140899:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6497 00000100:00080000:3.0:1788574747.140914:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574747.140924:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574747.140971:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864226944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574747.141003:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574747.141008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574747.141011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574747.141017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6497 00000100:00080000:3.0:1788574747.141023:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574747.141028:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574747.141040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574747.141049:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864226944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574747.141060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864227072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574747.141076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574747.141081:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574747.141083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574747.141089:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6497 00000100:00080000:3.0:1788574747.141095:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574747.141102:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574747.141110:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788574747.141140:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574747.141144:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864227200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574747.141159:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574747.141163:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574747.141165:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574747.141186:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574747/real 1788574747] req@ffff92fc67339180 x1875449864226944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574747.141202:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864226944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574747.141211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574747.141215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574747.141218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864226944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574747.141243:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6497 00000100:00100000:2.0:1788574747.141246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227072:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788574747.141266:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574747.141275:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788574747.141282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574747.141283:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574747.141295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574747/real 1788574747] req@ffff92fc67339f80 x1875449864227072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574747.141307:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864227328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574747.141312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864227072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574747.141357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574747.141361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574747.141364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574747.141368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574747.141382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574747.141391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574747/real 1788574747] req@ffff92fc6733ad80 x1875449864227200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574747.141399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864227200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574747.141404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574747.141406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574747.141408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227200:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788574747.141513:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574747.141563:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227328:192.168.202.132@tcp:8:kworker.0 00000100:00080000:3.0:1788574747.141566:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574747.141573:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574747.141582:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6497 00000100:00080000:3.0:1788574747.141592:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574747.141607:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574747.141617:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574747.141635:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864227456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574747.141644:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574747.141669:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574747/real 1788574747] req@ffff92fc67339180 x1875449864227328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574747.141684:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864227328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574747.141692:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574747.141695:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574747.141698:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574747.141714:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574747.141782:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574747.141793:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574747/real 1788574747] req@ffff92fc67339c00 x1875449864227456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574747.141802:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864227456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574747.141815:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574747.141817:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574747.141824:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574747.588927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574747.652806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574748.612879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574748.676825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574749.636855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574749.700830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574750.660842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574750.724821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574751.684860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574751.748812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574752.260815:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574752.260827:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574752.260832:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574752.260843:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6502 00000100:00080000:3.0:1788574752.260855:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574752.260864:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574752.260926:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864227584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574752.260955:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574752.260959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574752.260960:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574752.260963:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6502 00000100:00080000:3.0:1788574752.260966:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574752.260968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574752.260980:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574752.260989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864227712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574752.260998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574752.261002:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788574752.261004:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864227584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574752.261004:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574752.261012:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6502 00000100:00080000:3.0:1788574752.261015:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574752.261018:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574752.261021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574752.261030:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864227840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574752.261036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574752.261039:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574752.261040:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574752.261044:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6502 00000100:00080000:3.0:1788574752.261047:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574752.261049:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574752.261052:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574752.261060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864227968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574752.261065:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574752.261067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574752.261070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574752.261073:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6502 00000100:00080000:3.0:1788574752.261075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574752.261078:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574752.261081:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574752.261085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574752.261091:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864228096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574752.261135:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff92fc67339880 x1875449864227584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574752.261152:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864227584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574752.261162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574752.261165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574752.261168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864227584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574752.261197:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574752.261224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574752.261248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff92fc6733b800 x1875449864227712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574752.261258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864227712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574752.261265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574752.261267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574752.261270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574752.261275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574752.261290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574752.261299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff92fc67338a80 x1875449864227840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574752.261307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864227840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574752.261312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574752.261315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574752.261317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574752.261321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574752.261340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574752.261347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff92fc67338380 x1875449864227968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574752.261355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864227968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574752.261361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574752.261363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574752.261365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864227968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574752.261368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574752.261379:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574752.261385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574752/real 1788574752] req@ffff92fc6733b480 x1875449864228096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574752.261392:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864228096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574752.261398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574752.261400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574752.261403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574752.708920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574752.772887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574753.732935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574753.796892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574754.756872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574754.820870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574755.780839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574755.845094:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574756.804894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574756.868866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574757.380868:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574757.380880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574757.380884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574757.380896:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6507 00000100:00080000:3.0:1788574757.380908:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574757.380916:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574757.380966:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864228224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574757.381004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574757.381009:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574757.381011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574757.381016:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6507 00000100:00080000:3.0:1788574757.381020:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574757.381026:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574757.381039:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574757.381057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864228352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574757.381066:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864228224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574757.381076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574757.381080:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574757.381082:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574757.381086:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6507 00000100:00080000:3.0:1788574757.381089:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574757.381093:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574757.381098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574757.381108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864228480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574757.381117:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574757.381120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574757.381122:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574757.381125:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6507 00000100:00080000:3.0:1788574757.381128:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574757.381131:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574757.381136:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574757.381145:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864228608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574757.381147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574757.381154:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574757.381157:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574757.381159:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574757.381162:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6507 00000100:00080000:3.0:1788574757.381166:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574757.381170:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574757.381174:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574757.381183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864228736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574757.381202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff92fc6733aa00 x1875449864228224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574757.381221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864228224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574757.381233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574757.381238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574757.381242:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864228224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574757.381270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574757.381290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574757.381300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff92fc6733bb80 x1875449864228352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574757.381309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864228352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574757.381315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574757.381318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574757.381320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574757.381324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574757.381337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574757.381343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff92fc67338000 x1875449864228480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574757.381351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864228480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574757.381357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574757.381358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574757.381360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574757.381364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574757.381376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574757.381381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff92fc67338700 x1875449864228608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574757.381389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864228608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574757.381394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574757.381395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574757.381397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574757.381401:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574757.381412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574757.381418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574757/real 1788574757] req@ffff92fc67339500 x1875449864228736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574757.381426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864228736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574757.381431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574757.381432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574757.381434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574757.828876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574757.892830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574758.852906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574758.916813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574759.877574:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574759.940868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574760.900888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574760.964819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574761.924882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574761.988834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574762.500799:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574762.500808:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574762.500811:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574762.500819:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6513 00000100:00080000:3.0:1788574762.500829:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574762.500836:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574762.500869:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864228864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574762.500904:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574762.500907:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574762.500908:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574762.500911:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6513 00000100:00080000:3.0:1788574762.500914:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574762.500916:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574762.500925:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574762.500939:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864228992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574762.500948:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574762.500952:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574762.500954:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574762.500957:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6513 00000100:00080000:3.0:1788574762.500959:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574762.500965:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574762.500974:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864228864:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788574762.500974:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574762.500988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864229120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574762.500997:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574762.500999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574762.501001:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574762.501004:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6513 00000100:00080000:3.0:1788574762.501006:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574762.501008:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574762.501012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574762.501023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864229248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574762.501030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574762.501033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574762.501034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574762.501037:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6513 00000100:00080000:3.0:1788574762.501041:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574762.501044:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574762.501048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574762.501056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864229376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574762.501089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574762.501151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff92fc6733b100 x1875449864228864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574762.501173:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864228864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574762.501195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574762.501200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574762.501205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864228864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574762.501215:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574762.501243:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574762.501258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff92fc6733a680 x1875449864228992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574762.501273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864228992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574762.501291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574762.501295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574762.501299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864228992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574762.501337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574762.501360:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574762.501373:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff92fc67338e00 x1875449864229120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574762.501385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864229120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574762.501395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574762.501398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574762.501402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574762.501409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574762.501432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574762.501444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff92fc6733a300 x1875449864229248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574762.501457:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864229248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574762.501465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574762.501468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574762.501472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574762.501479:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574762.501498:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574762.501509:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574762/real 1788574762] req@ffff92fc67339500 x1875449864229376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574762.501520:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864229376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574762.501529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574762.501532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574762.501536:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574762.948971:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574763.012797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574763.972889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574764.036846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574764.996885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574765.060783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574766.020852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574766.084839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574767.044930:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574767.108830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574767.620909:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.620924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574767.620929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.620940:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6518 00000100:00080000:1.0:1788574767.620953:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.620972:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574767.621024:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864229504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574767.621081:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.621085:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574767.621087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.621092:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6518 00000100:00080000:1.0:1788574767.621097:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.621102:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574767.621116:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574767.621134:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864229632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574767.621146:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.621149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574767.621151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.621156:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6518 00000100:00080000:1.0:1788574767.621161:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574767.621161:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864229504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574767.621166:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574767.621173:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574767.621184:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864229760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574767.621191:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.621193:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574767.621195:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.621198:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6518 00000100:00080000:1.0:1788574767.621200:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.621202:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574767.621205:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574767.621214:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864229888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574767.621220:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574767.621223:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574767.621225:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574767.621227:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6518 00000100:00080000:1.0:1788574767.621230:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574767.621232:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574767.621235:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574767.621242:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864230016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574767.621260:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574767.621319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff92fc6728ad80 x1875449864229504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574767.621342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864229504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574767.621355:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574767.621360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574767.621365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864229504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574767.621397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574767.621449:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574767.621464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff92fc67289880 x1875449864229632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574767.621478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864229632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574767.621488:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574767.621491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574767.621495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574767.621502:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574767.621527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574767.621540:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff92fc67289f80 x1875449864229760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574767.621559:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864229760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574767.621568:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574767.621571:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574767.621574:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574767.621581:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574767.621603:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574767.621615:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff92fc67288000 x1875449864229888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574767.621628:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864229888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574767.621636:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574767.621639:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574767.621643:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864229888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574767.621649:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574767.621687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574767.621699:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff92fc67288e00 x1875449864230016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574767.621710:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864230016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574767.621767:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574767.621772:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574767.621775:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574768.068870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574768.132855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574769.092943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574769.156887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574770.116863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574770.180878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574771.140956:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574771.204848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574772.164876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574772.228866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574772.740868:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574772.740880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574772.740884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574772.740895:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6523 00000100:00080000:1.0:1788574772.740908:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574772.740915:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574772.740961:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864230144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574772.740990:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574772.740993:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574772.740995:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574772.740997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6523 00000100:00080000:1.0:1788574772.740999:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574772.741001:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574772.741011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574772.741020:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864230272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574772.741027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574772.741029:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574772.741030:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574772.741032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6523 00000100:00080000:1.0:1788574772.741034:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574772.741036:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574772.741038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574772.741044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864230400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574772.741050:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574772.741053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574772.741054:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574772.741057:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6523 00000100:00080000:1.0:1788574772.741058:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574772.741060:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574772.741063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574772.741070:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864230528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574772.741076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574772.741078:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864230144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574772.741079:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574772.741082:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574772.741087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6523 00000100:00080000:1.0:1788574772.741090:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574772.741092:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574772.741096:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574772.741104:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864230656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574772.741156:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574772.741204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff92fc6728a300 x1875449864230144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574772.741219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864230144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574772.741227:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574772.741230:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574772.741233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864230144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574772.741238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574772.741250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574772.741273:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff92fc6728b480 x1875449864230272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574772.741281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864230272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574772.741287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574772.741290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574772.741292:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574772.741295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574772.741308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574772.741314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff92fc6728aa00 x1875449864230400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574772.741320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864230400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574772.741325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574772.741327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574772.741329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574772.741354:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574772.741365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574772.741370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff92fc67288e00 x1875449864230528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574772.741377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864230528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574772.741382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574772.741384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574772.741386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574772.741396:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574772.741407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574772.741412:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff92fc67288000 x1875449864230656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574772.741419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864230656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574772.741424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574772.741425:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574772.741426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574773.188946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574773.252903:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574774.212938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574774.276859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574775.236910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574775.300840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574776.260861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574776.324812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574777.284950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574777.348856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574777.860889:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574777.860898:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574777.860904:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574777.860914:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6528 00000100:00080000:3.0:1788574777.860925:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574777.860934:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574777.860985:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864230784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574777.861016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574777.861021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574777.861023:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574777.861027:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6528 00000100:00080000:3.0:1788574777.861031:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574777.861035:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574777.861048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574777.861061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864230912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574777.861069:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574777.861071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574777.861073:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574777.861078:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6528 00000100:00100000:2.0:1788574777.861080:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864230784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574777.861082:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574777.861088:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574777.861093:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574777.861124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864231040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574777.861131:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574777.861133:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574777.861135:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574777.861138:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6528 00000100:00080000:3.0:1788574777.861141:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574777.861143:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574777.861147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574777.861154:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864231168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574777.861160:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574777.861162:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574777.861163:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574777.861166:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6528 00000100:00080000:3.0:1788574777.861168:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574777.861170:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574777.861172:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574777.861179:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864231296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574777.861187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574777.861245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff92fc67338700 x1875449864230784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574777.861266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864230784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574777.861277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574777.861281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574777.861285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864230784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574777.861311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574777.861335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574777.861347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff92fc67338000 x1875449864230912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574777.861360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864230912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574777.861368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574777.861371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574777.861374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864230912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574777.861380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788574777.861399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574777.861411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff92fc6733bb80 x1875449864231040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574777.861423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864231040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574777.861431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574777.861434:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574777.861437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788574777.861443:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574777.861460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574777.861469:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff92fc6733aa00 x1875449864231168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574777.861480:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864231168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574777.861488:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574777.861491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574777.861494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788574777.861500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788574777.861517:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574777.861528:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff92fc6733b480 x1875449864231296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574777.861538:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864231296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574777.861546:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574777.861549:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574777.861552:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574778.308920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574778.372873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574779.332940:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574779.396817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574780.356903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574780.420851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574781.380946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574781.444847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574782.404918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574782.468803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574782.980869:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574782.980880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574782.980885:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574782.980897:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6533 00000100:00080000:1.0:1788574782.980912:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574782.980923:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574782.980989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864231424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574782.981025:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574782.981029:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574782.981031:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574782.981039:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6533 00000100:00080000:1.0:1788574782.981044:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574782.981049:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574782.981063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574782.981073:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864231424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574782.981085:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864231552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574782.981099:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574782.981105:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574782.981107:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574782.981110:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6533 00000100:00080000:1.0:1788574782.981115:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574782.981120:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574782.981128:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574782.981141:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864231680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574782.981151:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574782.981153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574782.981166:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574782.981169:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574782.981176:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6533 00000100:00080000:1.0:1788574782.981180:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574782.981185:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574782.981191:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574782.981204:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864231808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574782.981208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff92fc67289f80 x1875449864231424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.981223:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574782.981235:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574782.981239:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574782.981248:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6533 00000100:00080000:1.0:1788574782.981252:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574782.981255:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574782.981261:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574782.981268:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864231936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574782.981341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864231424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574782.981481:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574782.981489:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574782.981549:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864231424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574782.981584:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574782.981605:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574782.981614:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff92fc67289880 x1875449864231552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574782.981625:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864231552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574782.981630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574782.981633:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574782.981635:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574782.981639:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574782.981651:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574782.981658:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff92fc6728ad80 x1875449864231680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574782.981664:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864231680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574782.981669:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574782.981671:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574782.981673:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574782.981677:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574782.981686:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574782.981691:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff92fc6728b800 x1875449864231808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574782.981700:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864231808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574782.981707:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574782.981710:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574782.981713:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574782.981719:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574782.981772:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574782.981781:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff92fc6728b100 x1875449864231936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574782.981789:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864231936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574782.981798:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574782.981803:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574782.981809:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864231936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574783.429013:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574783.492888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574784.452952:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574784.516842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574785.476887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574785.540829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574786.500858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574786.564819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574787.524922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574787.588893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574788.100842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574788.100849:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574788.100853:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574788.100864:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6538 00000100:00080000:1.0:1788574788.100874:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574788.100881:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574788.100917:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864232064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574788.100939:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574788.100944:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574788.100945:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574788.100949:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6538 00000100:00080000:1.0:1788574788.100953:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574788.100957:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574788.100966:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574788.100979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864232192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574788.100985:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864232064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574788.100995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574788.100999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574788.101000:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574788.101004:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6538 00000100:00080000:1.0:1788574788.101009:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574788.101013:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574788.101018:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574788.101027:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864232320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574788.101033:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574788.101035:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574788.101036:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574788.101040:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6538 00000800:00000100:3.0:1788574788.101041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574788.101044:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574788.101048:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574788.101051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574788.101058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864232448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574788.101064:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574788.101066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574788.101067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574788.101069:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6538 00000100:00080000:1.0:1788574788.101071:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574788.101074:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574788/real 1788574788] req@ffff92fc6728a680 x1875449864232064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574788.101076:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574788.101082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574788.101090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864232576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574788.101100:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864232064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574788.101112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574788.101117:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574788.101121:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864232064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574788.101141:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574788.101156:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574788.101164:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574788/real 1788574788] req@ffff92fc67289c00 x1875449864232192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574788.101171:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864232192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574788.101176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574788.101178:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574788.101180:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574788.101184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574788.101194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574788.101202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574788/real 1788574788] req@ffff92fc67288380 x1875449864232320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574788.101213:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864232320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574788.101219:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574788.101222:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574788.101225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574788.101230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574788.101248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574788.101257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574788/real 1788574788] req@ffff92fc67288a80 x1875449864232448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574788.101267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864232448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574788.101274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574788.101283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574788.101287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574788.101292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574788.101311:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574788.101322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574788/real 1788574788] req@ffff92fc67288700 x1875449864232576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574788.101333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864232576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574788.101341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574788.101343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574788.101346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574788.548879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574788.612843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574789.572909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574789.636844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574790.596849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574790.660842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574791.620841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574791.684837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574792.644895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574792.708834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574793.220807:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574793.220821:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574793.220825:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574793.220833:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6543 00000100:00080000:2.0:1788574793.220842:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574793.220848:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574793.220879:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864232704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574793.220900:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574793.220903:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574793.220904:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574793.220907:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6543 00000100:00080000:2.0:1788574793.220909:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574793.220911:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574793.220919:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574793.220927:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864232832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574793.220936:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574793.220939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574793.220941:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574793.220943:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6543 00000100:00080000:2.0:1788574793.220945:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574793.220948:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574793.220951:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574793.220957:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864232960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574793.220969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864232704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574793.220970:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574793.220982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574793.220988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574793.220992:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6543 00000100:00080000:2.0:1788574793.220995:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574793.220997:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574793.221000:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574793.221013:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864233088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574793.221020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574793.221022:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574793.221023:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574793.221027:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6543 00000100:00080000:2.0:1788574793.221031:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574793.221035:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574793.221041:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574793.221054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864233216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574793.221103:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574793.221147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574793.221173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574793.221191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574793.668946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574793.732899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574794.692917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574794.756889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574795.716886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574795.780824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788574796.292994:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788574796.293033:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574796.293050:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574796.293087:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574796.293099:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574796.293108:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788574796.293119:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574796.293143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff92fc4617e680 x1875449864232960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.293172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864232960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.293185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.293190:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574796.293194:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574796.293203:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff92fc4617c000 x1875449864233088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.293225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864233088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.293233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.293236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574796.293239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574796.293248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff92fc4617fb80 x1875449864233216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.293260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864233216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.293267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.293270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574796.293272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233216:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574796.293278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff92fc4617c380 x1875449864232704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.293289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864232704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.293296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.293300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574796.293303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864232704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574796.293310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574793/real 1788574796] req@ffff92fc4617d180 x1875449864232832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574796.293322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864232832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574796.293330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574796.293333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574796.293336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864232832:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788574796.740878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574796.804833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574797.764891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574797.828822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574798.340876:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574798.340887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574798.340891:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574798.340899:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6548 00000100:00080000:1.0:1788574798.340907:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574798.340914:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574798.340947:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864233344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574798.340978:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574798.340981:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574798.340984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574798.340988:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6548 00000100:00080000:1.0:1788574798.340992:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574798.340996:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574798.341007:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574798.341021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864233472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574798.341035:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574798.341036:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864233344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574798.341041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574798.341046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574798.341052:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6548 00000100:00080000:1.0:1788574798.341056:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574798.341061:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574798.341066:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574798.341081:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864233600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574798.341092:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574798.341096:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574798.341098:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574798.341102:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6548 00000100:00080000:1.0:1788574798.341106:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574798.341115:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574798.341120:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788574798.341123:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574798.341138:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864233728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574798.341148:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574798.341151:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574798.341153:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574798.341158:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6548 00000100:00080000:1.0:1788574798.341163:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574798.341167:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574798.341172:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff92fc6728bb80 x1875449864233344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788574798.341175:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574798.341193:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864233856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574798.341201:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864233344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.341216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.341221:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574798.341225:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864233344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574798.341254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574798.341280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574798.341296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff92fc67289500 x1875449864233472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.341309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864233472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.341326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.341330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574798.341333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574798.341340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574798.341362:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574798.341377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff92fc67289180 x1875449864233600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.341388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864233600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.341397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.341400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574798.341403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574798.341409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574798.341431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574798.341443:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff92fc67288700 x1875449864233728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.341456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864233728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.341463:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.341466:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574798.341470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574798.341489:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574798.341521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574798.341532:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574798/real 1788574798] req@ffff92fc67288a80 x1875449864233856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574798.341543:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864233856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574798.341558:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574798.341561:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574798.341564:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864233856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574798.788852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574798.852833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574799.812911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574799.876830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574800.836922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574800.900911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574801.860941:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574801.924817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574802.884865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574802.948955:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574803.460836:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.460844:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574803.460848:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.460858:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6554 00000100:00080000:1.0:1788574803.460869:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.460877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574803.460915:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864233984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574803.460947:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.460952:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574803.460954:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.460959:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6554 00000100:00080000:1.0:1788574803.460963:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.460968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574803.460977:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574803.460988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864234112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574803.460997:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.461001:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574803.461003:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.461008:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6554 00000100:00100000:3.0:1788574803.461009:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864233984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574803.461011:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.461017:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574803.461021:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574803.461034:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864234240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574803.461044:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.461048:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574803.461050:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.461055:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6554 00000100:00080000:1.0:1788574803.461060:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.461064:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574803.461070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574803.461076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574803.461084:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864234368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574803.461091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574803.461094:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574803.461096:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574803.461099:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6554 00000100:00080000:1.0:1788574803.461102:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574803.461104:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574803.461108:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574803.461116:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864234496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574803.461128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff92fc67288380 x1875449864233984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.461145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864233984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.461154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.461157:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574803.461160:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864233984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574803.461165:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574803.461183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574803.461193:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff92fc67289c00 x1875449864234112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.461201:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864234112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.461207:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.461209:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574803.461212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574803.461231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574803.461242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574803.461249:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff92fc6728a680 x1875449864234240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.461256:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864234240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.461262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.461263:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574803.461265:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574803.461269:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574803.461280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574803.461286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff92fc6728b100 x1875449864234368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.461293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864234368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.461299:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.461301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574803.461303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574803.461306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574803.461332:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574803.461341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574803/real 1788574803] req@ffff92fc6728b800 x1875449864234496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574803.461349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864234496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574803.461354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574803.461356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574803.461364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574803.908928:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574803.972809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574804.932867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574804.996843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574805.957012:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574806.020876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574806.980879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574807.044843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574808.004947:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574808.068798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574808.580891:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.580908:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574808.580912:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.580922:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6559 00000100:00080000:0.0:1788574808.580931:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.580938:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574808.580972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864234624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.580998:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.581001:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.581003:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.581006:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6559 00000100:00080000:0.0:1788574808.581009:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.581013:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574808.581021:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574808.581032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864234752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.581042:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.581047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574808.581050:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.581054:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6559 00000100:00080000:0.0:1788574808.581056:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.581060:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574808.581066:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574808.581077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864234880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.581092:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574808.581092:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864234624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574808.581103:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.581110:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.581116:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6559 00000100:00080000:0.0:1788574808.581120:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.581123:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574808.581126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574808.581141:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864235008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.581150:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.581154:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574808.581156:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.581172:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6559 00000100:00080000:0.0:1788574808.581176:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.581181:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574808.581187:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574808.581198:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864235136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574808.581226:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574808.581286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff92fc6704b100 x1875449864234624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.581308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864234624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.581321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.581326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574808.581331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864234624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574808.581347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574808.581371:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574808.581385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff92fc67049880 x1875449864234752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.581399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864234752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.581408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.581411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574808.581415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574808.581445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574808.581467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574808.581477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff92fc6704ad80 x1875449864234880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.581489:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864234880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.581497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.581500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574808.581504:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864234880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574808.581510:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574808.581539:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574808.581550:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff92fc6704a680 x1875449864235008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.581562:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864235008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.581570:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.581573:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574808.581576:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574808.581582:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574808.581601:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574808.581621:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574808] req@ffff92fc6704aa00 x1875449864235136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574808.581633:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864235136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574808.581641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574808.581654:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574808.581657:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574809.028829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574809.092839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574810.052992:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574810.116822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574811.076870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574811.140882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574812.100925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574812.164839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574813.124932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574813.188865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574813.700900:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.700915:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574813.700921:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.700934:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6564 00000100:00080000:0.0:1788574813.700948:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.700957:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574813.701023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864235264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.701058:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.701062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574813.701064:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.701068:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6564 00000100:00080000:0.0:1788574813.701072:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.701076:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574813.701088:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574813.701102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864235392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.701114:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.701118:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574813.701120:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574813.701125:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864235264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574813.701127:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6564 00000100:00080000:0.0:1788574813.701134:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.701139:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574813.701144:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574813.701156:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864235520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.701166:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.701169:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574813.701171:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.701174:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6564 00000100:00080000:0.0:1788574813.701178:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.701181:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574813.701187:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574813.701197:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864235648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574813.701205:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574813.701209:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574813.701210:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574813.701214:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6564 00000100:00080000:0.0:1788574813.701218:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574813.701224:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574813.701227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574813.701231:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574813.701246:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864235776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574813.701292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff92fc6704a300 x1875449864235264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.701315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864235264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.701333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.701338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574813.701342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864235264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574813.701372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574813.701398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574813.701411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff92fc67048000 x1875449864235392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.701423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864235392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.701432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.701436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574813.701439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574813.701446:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574813.701466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574813.701477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff92fc67049f80 x1875449864235520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.701488:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864235520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.701497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.701500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574813.701503:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574813.701510:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574813.701531:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574813.701542:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff92fc67048700 x1875449864235648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.701554:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864235648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.701562:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.701566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574813.701569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574813.701575:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574813.701595:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574813.701607:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff92fc67048380 x1875449864235776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574813.701618:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864235776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574813.701626:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574813.701630:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574813.701633:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864235776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574814.148967:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574814.212837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574815.172896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574815.236836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574816.196888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574816.260865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574817.220975:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574817.284839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574818.244931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574818.308855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574818.820817:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.820826:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574818.820830:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.820839:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6569 00000100:00080000:0.0:1788574818.820849:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.820857:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574818.820906:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864235904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574818.820936:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.820940:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574818.820942:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.820946:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6569 00000100:00080000:0.0:1788574818.820950:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.820954:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574818.820963:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574818.820975:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864236032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574818.820979:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864235904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574818.820986:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.820989:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574818.820990:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.820993:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6569 00000100:00080000:0.0:1788574818.820996:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.820998:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574818.821001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574818.821009:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864236160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574818.821015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.821017:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574818.821018:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.821020:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6569 00000100:00080000:0.0:1788574818.821023:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.821025:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574818.821028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574818.821034:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864236288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574818.821040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574818.821042:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574818.821046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574818.821047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574818.821050:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6569 00000100:00080000:0.0:1788574818.821053:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574818.821055:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574818.821058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574818.821064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864236416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574818.821084:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff92fc67048a80 x1875449864235904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.821105:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864235904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.821113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.821116:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574818.821119:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864235904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574818.821138:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574818.821158:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574818.821167:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff92fc67048380 x1875449864236032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.821176:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864236032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.821181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.821183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574818.821186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574818.821194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574818.821216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574818.821240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff92fc67048700 x1875449864236160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.821256:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864236160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.821269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.821325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574818.821332:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574818.821339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574818.821360:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574818.821486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff92fc67049f80 x1875449864236288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.821495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864236288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.821501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.821503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574818.821506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574818.821510:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574818.821527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574818.821588:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff92fc67048000 x1875449864236416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574818.821605:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864236416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574818.821610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574818.821612:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574818.821614:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574819.268893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574819.332816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574820.292891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574820.356823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574821.316960:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574821.380869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574822.340896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574822.404856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574823.364965:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574823.428806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574823.940849:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574823.940857:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574823.940861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574823.940868:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6574 00000100:00080000:0.0:1788574823.940877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574823.940883:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574823.940916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864236544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574823.940940:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574823.940943:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574823.940944:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574823.940948:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6574 00000100:00080000:0.0:1788574823.940951:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574823.940955:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574823.940967:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574823.940981:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864236672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574823.940991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864236544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574823.940994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574823.941000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574823.941002:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574823.941006:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6574 00000100:00080000:0.0:1788574823.941011:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574823.941015:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574823.941021:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574823.941032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864236800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574823.941039:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574823.941041:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574823.941043:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574823.941045:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6574 00000100:00080000:0.0:1788574823.941047:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574823.941049:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574823.941053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574823.941057:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574823.941065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864236928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574823.941073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574823.941077:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574823.941080:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574823.941091:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6574 00000100:00080000:0.0:1788574823.941097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574823.941101:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574823.941107:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574823.941114:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff92fc6704a300 x1875449864236544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574823.941123:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864237056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574823.941134:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864236544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.941143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.941146:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574823.941150:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864236544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574823.941169:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574823.941186:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574823.941195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff92fc6704aa00 x1875449864236672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.941203:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864236672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.941209:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.941211:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574823.941213:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574823.941217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574823.941227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574823.941234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff92fc6704a680 x1875449864236800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.941242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864236800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.941247:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.941249:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574823.941251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574823.941254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574823.941266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574823.941272:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff92fc6704ad80 x1875449864236928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.941279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864236928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.941284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.941287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574823.941289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864236928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574823.941292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574823.941301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574823.941307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff92fc67049880 x1875449864237056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574823.941314:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864237056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574823.941320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574823.941322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574823.941324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574824.388910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574824.452883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574825.412926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574825.476837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574826.436895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574826.500851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574827.460895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574827.524790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574828.484890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574828.548826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574829.060884:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574829.060895:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574829.060900:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574829.060911:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6579 00000100:00080000:2.0:1788574829.060922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574829.060930:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574829.060977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864237184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574829.061011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574829.061015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574829.061017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574829.061022:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6579 00000100:00080000:2.0:1788574829.061026:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574829.061029:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574829.061040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574829.061052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864237312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574829.061060:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574829.061062:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574829.061064:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574829.061067:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6579 00000100:00080000:2.0:1788574829.061069:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574829.061073:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864237184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574829.061074:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574829.061078:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574829.061087:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864237440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574829.061093:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574829.061095:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574829.061096:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574829.061098:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6579 00000100:00080000:2.0:1788574829.061100:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574829.061102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574829.061105:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574829.061112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864237568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574829.061118:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574829.061120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574829.061122:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574829.061123:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6579 00000100:00080000:2.0:1788574829.061140:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574829.061142:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574829.061145:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574829.061152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864237696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574829.061162:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574829.061214:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574829/real 1788574829] req@ffff92fc4617f100 x1875449864237184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574829.061230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864237184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574829.061239:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574829.061242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574829.061245:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864237184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574829.061267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574829.061286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574829.061298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574829/real 1788574829] req@ffff92fc4617f800 x1875449864237312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574829.061309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864237312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574829.061319:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574829.061322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574829.061326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574829.061332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574829.061354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574829.061365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574829/real 1788574829] req@ffff92fc4617ca80 x1875449864237440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574829.061377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864237440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574829.061386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574829.061389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574829.061392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574829.061396:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574829.061411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574829.061418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574829/real 1788574829] req@ffff92fc4617c700 x1875449864237568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574829.061425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864237568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574829.061430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574829.061432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574829.061434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574829.061437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574829.061446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574829.061451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574829/real 1788574829] req@ffff92fc4617dc00 x1875449864237696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574829.061459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864237696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574829.061464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574829.061465:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574829.061467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574829.508955:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574829.572869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574830.532867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574830.596827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574831.556919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574831.620842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574832.580871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574832.644833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574833.604892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574833.668824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574834.180885:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.180896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574834.180901:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.180911:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6584 00000100:00080000:0.0:1788574834.180923:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.180931:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574834.180978:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864237824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.181011:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.181016:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574834.181017:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.181020:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6584 00000100:00080000:0.0:1788574834.181023:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.181026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574834.181036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574834.181055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864237952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.181066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.181069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574834.181071:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574834.181076:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864237824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574834.181077:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6584 00000100:00080000:0.0:1788574834.181083:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.181089:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574834.181095:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574834.181115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864238080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.181125:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574834.181129:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574834.181131:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.181135:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6584 00000100:00080000:0.0:1788574834.181139:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.181143:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574834.181150:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574834.181161:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864238208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574834.181169:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574834.181170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574834.181173:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574834.181176:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574834.181179:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6584 00000100:00080000:0.0:1788574834.181182:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574834.181185:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574834.181189:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574834.181198:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864238336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574834.181236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff92fc6704b100 x1875449864237824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.181256:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864237824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.181268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.181272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574834.181276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864237824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574834.181305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574834.181328:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574834.181340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff92fc6704bb80 x1875449864237952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.181352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864237952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.181366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.181369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574834.181372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864237952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574834.181378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574834.181396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574834.181407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff92fc67049500 x1875449864238080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.181417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864238080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.181425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.181428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574834.181431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574834.181438:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574834.181458:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574834.181467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff92fc67049180 x1875449864238208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.181478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864238208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.181485:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.181488:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574834.181492:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574834.181513:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574834.181534:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574834.181553:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574834/real 1788574834] req@ffff92fc67049c00 x1875449864238336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574834.181565:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864238336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574834.181572:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574834.181576:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574834.181580:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574834.628999:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574834.692902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574835.652961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574835.716823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574836.676924:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574836.740830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574837.700917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574837.764831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574838.724915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574838.788810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574839.300852:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574839.300861:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574839.300865:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574839.300873:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6589 00000100:00080000:1.0:1788574839.300883:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574839.300889:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574839.300922:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864238464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574839.300958:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574839.300961:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574839.300963:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574839.300966:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6589 00000100:00080000:1.0:1788574839.300968:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574839.300971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574839.300979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574839.300988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864238592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574839.300994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574839.300997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574839.300998:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574839.301000:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6589 00000100:00080000:1.0:1788574839.301002:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574839.301005:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574839.301008:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574839.301015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864238720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574839.301023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574839.301026:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574839.301027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574839.301033:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6589 00000100:00080000:1.0:1788574839.301035:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574839.301035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864238464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574839.301042:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574839.301048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574839.301054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864238848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574839.301060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574839.301062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574839.301063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574839.301067:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6589 00000100:00080000:1.0:1788574839.301071:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574839.301075:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574839.301081:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574839.301093:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864238976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574839.301102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574839.301146:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff92fc6728ad80 x1875449864238464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.301167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864238464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.301179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.301183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574839.301187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864238464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574839.301195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574839.301220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574839.301237:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff92fc67289880 x1875449864238592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.301251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864238592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.301261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.301265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574839.301268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574839.301358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574839.301425:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574839.301438:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff92fc67289f80 x1875449864238720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.301447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864238720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.301453:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.301455:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574839.301458:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574839.301464:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574839.301486:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574839.301498:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff92fc67288000 x1875449864238848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.301512:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864238848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.301521:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.301523:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574839.301525:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574839.301530:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574839.301545:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574839.301553:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574839/real 1788574839] req@ffff92fc67288e00 x1875449864238976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574839.301561:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864238976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574839.301566:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574839.301568:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574839.301570:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864238976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574839.748884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574839.812834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574840.772936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574840.836818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574841.797012:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574841.860850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574842.820942:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574842.884808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574843.844912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574843.908840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574844.420898:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.420913:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574844.420919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.420937:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6595 00000100:00080000:0.0:1788574844.420952:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.420963:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574844.421018:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864239104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.421044:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.421047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574844.421048:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.421051:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6595 00000100:00080000:0.0:1788574844.421054:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.421057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574844.421067:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574844.421079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864239232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.421086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.421090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574844.421091:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864239104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574844.421092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.421099:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6595 00000100:00080000:0.0:1788574844.421101:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.421104:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574844.421106:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574844.421113:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864239360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.421118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.421121:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574844.421122:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.421125:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6595 00000100:00080000:0.0:1788574844.421127:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.421129:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574844.421132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574844.421139:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864239488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574844.421145:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574844.421147:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574844.421148:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574844.421151:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6595 00000100:00080000:0.0:1788574844.421154:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574844.421156:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574844.421158:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574844.421165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864239616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574844.421180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574844.421254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff92fc67048e00 x1875449864239104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.421276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864239104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.421289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.421294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574844.421298:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864239104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574844.421326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574844.421353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574844.421367:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff92fc67049c00 x1875449864239232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.421378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864239232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.421386:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.421389:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574844.421392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574844.421397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574844.421415:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574844.421427:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff92fc67049180 x1875449864239360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.421437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864239360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.421444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.421447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574844.421450:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574844.421455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574844.421475:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574844.421486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff92fc67049500 x1875449864239488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.421498:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864239488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.421507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.421510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574844.421513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574844.421519:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574844.421538:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574844.421549:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574844/real 1788574844] req@ffff92fc6704bb80 x1875449864239616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574844.421561:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864239616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574844.421568:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574844.421571:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574844.421574:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574844.868910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574844.932836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574845.892929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574845.956835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574846.916914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574846.980846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574847.940898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574848.004837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574848.964967:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574849.028889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574849.540806:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.540818:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574849.540824:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.540836:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6600 00000100:00080000:0.0:1788574849.540848:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.540857:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574849.540899:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864239744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.540926:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.540929:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574849.540930:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.540933:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6600 00000100:00080000:0.0:1788574849.540935:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.540937:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574849.540947:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574849.540958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864239872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.540969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.540972:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574849.540974:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574849.540979:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864239744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574849.540981:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6600 00000100:00080000:0.0:1788574849.540989:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.540993:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574849.540998:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574849.541012:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864240000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.541028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.541031:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574849.541033:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.541038:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6600 00000100:00080000:0.0:1788574849.541042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.541047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574849.541053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574849.541062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574849.541064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864240128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574849.541076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574849.541079:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574849.541080:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574849.541083:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6600 00000100:00080000:0.0:1788574849.541087:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574849.541089:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574849.541093:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574849.541101:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864240256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574849.541142:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff92fc6704b100 x1875449864239744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.541159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864239744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.541178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.541181:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574849.541184:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864239744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574849.541207:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574849.541226:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574849.541235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff92fc67049880 x1875449864239872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.541242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864239872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.541248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.541250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574849.541252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864239872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574849.541255:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574849.541264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574849.541270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff92fc6704ad80 x1875449864240000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.541277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864240000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.541282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.541283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574849.541285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574849.541289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574849.541300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574849.541305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff92fc6704a680 x1875449864240128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.541312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864240128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.541316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.541318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574849.541319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574849.541323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574849.541332:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574849.541337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574849/real 1788574849] req@ffff92fc6704aa00 x1875449864240256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574849.541344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864240256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574849.541348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574849.541349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574849.541351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574849.988905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574850.052875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574851.012966:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574851.076878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574852.037004:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574852.100832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574853.061042:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574853.124901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574854.084960:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574854.148858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574854.661029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574854.661039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574854.661043:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574854.661052:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6605 00000100:00080000:2.0:1788574854.661062:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574854.661068:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574854.661100:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864240384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574854.661124:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574854.661147:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574854.661149:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574854.661153:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6605 00000100:00080000:2.0:1788574854.661157:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574854.661160:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574854.661169:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574854.661180:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864240512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574854.661190:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574854.661196:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574854.661199:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574854.661199:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864240384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574854.661216:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6605 00000100:00080000:2.0:1788574854.661219:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574854.661222:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574854.661225:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574854.661236:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864240640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574854.661244:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574854.661246:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574854.661247:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574854.661250:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6605 00000100:00080000:2.0:1788574854.661253:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574854.661255:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574854.661258:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574854.661266:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864240768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574854.661272:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574854.661274:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574854.661275:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574854.661279:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6605 00000100:00080000:2.0:1788574854.661283:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574854.661286:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574854.661289:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574854.661297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574854.661302:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864240896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574854.661359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff92fc4617ed80 x1875449864240384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.661390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864240384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.661400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.661406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574854.661412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864240384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574854.661439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574854.661465:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574854.661477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff92fc4617f480 x1875449864240512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.661495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864240512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.661503:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.661506:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574854.661509:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574854.661514:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574854.661532:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574854.661538:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff92fc4617dc00 x1875449864240640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.661546:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864240640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.661556:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.661558:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574854.661561:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574854.661564:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574854.661575:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574854.661581:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff92fc4617c700 x1875449864240768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.661589:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864240768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.661594:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.661596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574854.661598:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574854.661601:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574854.661612:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574854.661619:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff92fc4617ca80 x1875449864240896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.661626:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864240896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.661631:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574854.661634:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574854.661638:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864240896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574855.108919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574855.172856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574856.132935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574856.196839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574857.156892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574857.220844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574858.180933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574858.244833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574859.204918:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574859.268828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574859.780881:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.780893:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574859.780898:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.780909:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6610 00000100:00080000:0.0:1788574859.780921:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.780930:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574859.780984:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864241024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.781012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.781015:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574859.781017:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.781020:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6610 00000100:00080000:0.0:1788574859.781023:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.781026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574859.781036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574859.781048:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864241152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.781062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.781067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574859.781067:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864241024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574859.781069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.781077:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6610 00000100:00080000:0.0:1788574859.781080:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.781082:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574859.781085:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574859.781096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864241280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.781103:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.781106:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574859.781107:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.781110:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6610 00000100:00080000:0.0:1788574859.781112:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.781115:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574859.781118:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574859.781126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864241408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.781131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.781134:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574859.781135:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.781138:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6610 00000100:00080000:0.0:1788574859.781142:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.781144:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574859.781148:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574859.781157:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864241536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574859.781202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574859.781228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574859.781245:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574859.781261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574860.228995:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574860.292840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574861.252845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574861.316852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574862.276935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574862.340880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574862.853030:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574862.853076:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574862.853092:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574862.853136:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574862.853146:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574862.853155:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574862.853164:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574862.853198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff92fc67048000 x1875449864241152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.853228:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864241152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.853241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.853246:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574862.853250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574862.853258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff92fc67049f80 x1875449864241280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.853271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864241280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.853279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.853283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574862.853287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574862.853295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff92fc67048700 x1875449864241408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.853308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864241408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.853315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.853319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574862.853323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574862.853330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff92fc67048380 x1875449864241536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.853342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864241536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.853349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.853353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574862.853356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241536:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574862.853381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574862] req@ffff92fc6704a300 x1875449864241024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574862.853396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864241024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574862.853404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574862.853408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574862.853411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864241024:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788574863.300849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574863.364848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574864.324887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574864.388874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574864.900930:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574864.900941:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574864.900945:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574864.900957:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6615 00000100:00080000:2.0:1788574864.900968:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574864.900976:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574864.901016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864241664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574864.901047:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574864.901050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574864.901051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574864.901054:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6615 00000100:00080000:2.0:1788574864.901058:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574864.901060:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574864.901072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574864.901090:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864241792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574864.901105:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574864.901105:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864241664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574864.901114:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574864.901119:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574864.901125:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6615 00000100:00080000:2.0:1788574864.901129:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574864.901133:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574864.901139:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574864.901154:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864241920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574864.901166:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574864.901170:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574864.901172:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574864.901177:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6615 00000100:00080000:2.0:1788574864.901199:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574864.901205:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574864.901212:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574864.901223:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574864.901229:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864242048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574864.901250:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574864.901254:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574864.901257:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574864.901262:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6615 00000100:00080000:2.0:1788574864.901267:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574864.901272:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574864.901278:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574864.901292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff92fc4617f800 x1875449864241664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574864.901295:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864242176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574864.901347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864241664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.901362:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.901368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574864.901373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864241664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574864.901416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574864.901448:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574864.901464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff92fc4617f100 x1875449864241792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.901478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864241792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.901487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.901491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574864.901494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574864.901501:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574864.901524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574864.901538:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff92fc4617d180 x1875449864241920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.901553:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864241920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.901563:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.901566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574864.901570:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864241920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574864.901577:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574864.901603:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574864.901616:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff92fc4617c380 x1875449864242048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.901631:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864242048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.901641:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.901645:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574864.901648:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574864.901656:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574864.901682:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574864.901695:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff92fc4617fb80 x1875449864242176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574864.901708:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864242176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.901772:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574864.901777:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574864.901781:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574865.348854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574865.412840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574866.372931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574866.436822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574867.396822:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574867.460825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574868.420909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574868.484832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574869.444815:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574869.508827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574870.020841:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574870.020850:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574870.020854:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574870.020862:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6620 00000100:00080000:1.0:1788574870.020870:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574870.020877:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574870.020908:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864242304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574870.020935:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574870.020939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574870.020941:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574870.020945:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6620 00000100:00080000:1.0:1788574870.020948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574870.020953:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574870.020964:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574870.020979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864242432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574870.020990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574870.020994:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574870.020996:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574870.021001:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6620 00000100:00080000:1.0:1788574870.021012:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574870.021022:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574870.021029:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864242304:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788574870.021032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574870.021057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864242560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574870.021071:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574870.021075:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574870.021078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574870.021085:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6620 00000100:00080000:1.0:1788574870.021091:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574870.021096:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574870.021103:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574870.021119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864242688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574870.021134:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574870.021137:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574870.021142:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574870.021146:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574870.021154:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6620 00000100:00080000:1.0:1788574870.021159:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574870.021167:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574870.021174:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574870.021189:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864242816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574870.021204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574870/real 1788574870] req@ffff92fc6728aa00 x1875449864242304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574870.021231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864242304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574870.021241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574870.021245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574870.021248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864242304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574870.021255:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574870.021282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574870.021296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574870/real 1788574870] req@ffff92fc6728b480 x1875449864242432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574870.021310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864242432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574870.021320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574870.021324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574870.021327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574870.021355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574870.021372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574870.021380:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574870/real 1788574870] req@ffff92fc6728a300 x1875449864242560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574870.021388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864242560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574870.021394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574870.021396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574870.021398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574870.021402:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574870.021413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574870.021420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574870/real 1788574870] req@ffff92fc67288e00 x1875449864242688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574870.021427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864242688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574870.021432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574870.021433:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574870.021435:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574870.021439:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574870.021451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574870.021457:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574870/real 1788574870] req@ffff92fc67288000 x1875449864242816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574870.021465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864242816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574870.021470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574870.021472:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574870.021474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864242816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574870.468941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574870.532802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574871.492902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574871.556877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574872.516974:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574872.580902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574873.540895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574873.604888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574874.564908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574874.628859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574875.140885:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574875.140902:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574875.140908:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574875.140924:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6625 00000100:00080000:1.0:1788574875.140938:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574875.140949:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574875.141013:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864242944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574875.141052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574875.141057:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574875.141059:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574875.141063:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6625 00000100:00080000:1.0:1788574875.141068:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574875.141072:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574875.141084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574875.141098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864243072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574875.141109:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574875.141114:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574875.141115:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864242944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574875.141117:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574875.141124:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6625 00000100:00080000:1.0:1788574875.141129:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574875.141133:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574875.141138:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574875.141152:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864243200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574875.141167:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574875.141171:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574875.141173:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574875.141177:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6625 00000100:00080000:1.0:1788574875.141182:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574875.141186:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574875.141199:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574875.141203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574875.141215:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864243328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574875.141225:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574875.141230:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574875.141232:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574875.141236:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6625 00000100:00080000:1.0:1788574875.141240:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574875.141244:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574875.141250:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574875.141262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574875/real 1788574875] req@ffff92fc67289f80 x1875449864242944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574875.141273:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864243456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574875.141288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864242944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574875.141300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574875.141304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574875.141309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864242944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574875.141339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574875.141359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574875.141370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574875/real 1788574875] req@ffff92fc67289880 x1875449864243072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574875.141381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864243072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574875.141395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574875.141398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574875.141401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574875.141406:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574875.141423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574875.141432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574875/real 1788574875] req@ffff92fc6728ad80 x1875449864243200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574875.141442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864243200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574875.141449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574875.141452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574875.141456:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574875.141461:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574875.141491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574875.141503:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574875/real 1788574875] req@ffff92fc6728b800 x1875449864243328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574875.141515:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864243328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574875.141530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574875.141533:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574875.141537:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574875.141543:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574875.141561:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574875.141570:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574875/real 1788574875] req@ffff92fc6728b100 x1875449864243456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574875.141581:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864243456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574875.141588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574875.141591:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574875.141594:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574875.588883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574875.652790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574876.612866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574876.676830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574877.636865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574877.700829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574878.660892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574878.724825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574879.684861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574879.748798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574880.260833:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574880.260842:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574880.260847:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574880.260856:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6630 00000100:00080000:0.0:1788574880.260871:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574880.260878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574880.260915:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864243584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574880.260951:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574880.260955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574880.260957:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574880.260960:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6630 00000100:00080000:0.0:1788574880.260963:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574880.260967:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574880.260978:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574880.260991:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864243712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574880.261002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574880.261006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864243584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574880.261007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574880.261011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574880.261017:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6630 00000100:00080000:0.0:1788574880.261020:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574880.261024:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574880.261028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574880.261040:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864243840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574880.261048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574880.261051:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574880.261053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574880.261057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6630 00000100:00080000:0.0:1788574880.261060:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574880.261065:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574880.261065:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574880.261073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574880.261084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864243968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574880.261092:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574880.261096:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574880.261097:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574880.261104:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6630 00000100:00100000:3.0:1788574880.261104:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff92fc67048a80 x1875449864243584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574880.261107:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574880.261114:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574880.261122:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574880.261134:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864243584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574880.261135:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864244096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574880.261163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.261168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574880.261171:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864243584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574880.261196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574880.261220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574880.261230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff92fc6704a300 x1875449864243712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.261238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864243712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.261244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.261245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574880.261248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574880.261251:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574880.261263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574880.261269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff92fc67048380 x1875449864243840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.261275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864243840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.261280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.261282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574880.261283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574880.261286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574880.261296:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574880.261300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff92fc67048700 x1875449864243968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.261307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864243968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.261311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.261313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574880.261315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864243968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574880.261318:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574880.261326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574880.261331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574880/real 1788574880] req@ffff92fc67049f80 x1875449864244096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574880.261337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864244096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574880.261342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574880.261343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574880.261345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574880.708918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574880.772819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574881.732866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574881.796804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574882.756934:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574882.820857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574883.780895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574883.844898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574884.804943:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574884.868830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574885.380854:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574885.380862:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574885.380866:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574885.380874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6635 00000100:00080000:0.0:1788574885.380884:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574885.380891:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574885.380929:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864244224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574885.380962:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574885.380966:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574885.380968:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574885.380972:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6635 00000100:00080000:0.0:1788574885.380977:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574885.380981:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574885.380991:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574885.381006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864244352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574885.381016:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574885.381021:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574885.381024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574885.381028:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6635 00000100:00080000:0.0:1788574885.381032:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574885.381036:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864244224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574885.381038:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574885.381046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574885.381057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864244480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574885.381073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574885.381076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574885.381078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574885.381080:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6635 00000100:00080000:0.0:1788574885.381083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574885.381085:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574885.381088:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574885.381097:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864244608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574885.381102:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574885.381105:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574885.381107:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574885.381112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574885.381115:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6635 00000100:00080000:0.0:1788574885.381119:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574885.381121:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574885.381125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574885.381134:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864244736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574885.381162:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff92fc67048000 x1875449864244224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.381182:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864244224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.381194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.381199:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574885.381203:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864244224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574885.381212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574885.381235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574885.381250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff92fc6704aa00 x1875449864244352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.381274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864244352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.381283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.381287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574885.381291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574885.381315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574885.381335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574885.381347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff92fc6704a680 x1875449864244480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.381378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864244480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.381387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.381390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574885.381394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574885.381400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574885.381422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574885.381433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff92fc6704ad80 x1875449864244608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.381448:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864244608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.381457:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.381460:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574885.381464:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574885.381471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574885.381491:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574885.381503:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574885/real 1788574885] req@ffff92fc67049880 x1875449864244736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574885.381515:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864244736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574885.381524:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574885.381527:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574885.381530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574885.828880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574885.892831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574886.852901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574886.916832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574887.876906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574887.940872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574888.900842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574888.964850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574889.924863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574889.988834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574890.500842:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574890.500852:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574890.500856:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574890.500866:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6641 00000100:00080000:0.0:1788574890.500877:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574890.500885:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574890.500930:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864244864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574890.500957:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574890.500962:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574890.500964:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574890.500969:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6641 00000100:00080000:0.0:1788574890.500973:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574890.500978:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574890.500988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574890.501012:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864244992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574890.501028:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864244864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574890.501029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574890.501038:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574890.501041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574890.501047:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6641 00000100:00080000:0.0:1788574890.501051:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574890.501056:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574890.501062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574890.501075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864245120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574890.501085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574890.501089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574890.501091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574890.501099:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6641 00000800:00000100:1.0:1788574890.501100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574890.501104:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574890.501110:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574890.501116:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574890.501128:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864245248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574890.501137:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574890.501140:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574890.501143:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574890.501145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff92fc6704b100 x1875449864244864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574890.501167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864244864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574890.501176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574890.501179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574890.501182:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864244864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574890.501198:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6641 00000100:00080000:0.0:1788574890.501204:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574890.501206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244992:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788574890.501212:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574890.501219:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574890.501229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788574890.501237:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864245376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574890.501242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff92fc6704bb80 x1875449864244992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574890.501268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864244992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574890.501275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574890.501277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574890.501280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864244992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574890.501285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574890.501299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574890.501307:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff92fc67049500 x1875449864245120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574890.501332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864245120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574890.501346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574890.501350:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574890.501352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574890.501357:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574890.501374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574890.501383:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff92fc67049180 x1875449864245248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574890.501392:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864245248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574890.501398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574890.501400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574890.501402:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574890.501421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574890.501440:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574890.501462:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574890/real 1788574890] req@ffff92fc67049c00 x1875449864245376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574890.501478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864245376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574890.501487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574890.501491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574890.501496:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574890.948874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574891.012876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574891.972890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574892.036886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574892.996914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574893.060813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574894.020887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574894.084855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574895.044972:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574895.108802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574895.620863:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574895.620875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574895.620879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574895.620889:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6646 00000100:00080000:1.0:1788574895.620900:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574895.620906:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574895.620945:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864245504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574895.620969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574895.620972:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574895.620973:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574895.620976:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6646 00000100:00080000:1.0:1788574895.620979:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574895.620981:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574895.620988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574895.620997:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864245632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574895.621004:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574895.621006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574895.621007:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574895.621010:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6646 00000100:00080000:1.0:1788574895.621012:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574895.621015:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574895.621019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574895.621023:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864245504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574895.621034:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864245760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574895.621041:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574895.621043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574895.621045:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574895.621047:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6646 00000100:00080000:1.0:1788574895.621049:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574895.621051:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574895.621054:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574895.621064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864245888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574895.621070:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574895.621072:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574895.621074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574895.621076:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6646 00000100:00080000:1.0:1788574895.621084:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574895.621086:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574895.621089:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574895.621094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574895.621099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864246016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574895.621144:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff92fc6728a680 x1875449864245504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.621164:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864245504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.621180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.621185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574895.621189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864245504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574895.621198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574895.621221:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574895.621235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff92fc67289c00 x1875449864245632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.621248:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864245632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.621257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.621260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574895.621264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574895.621302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574895.621326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574895.621338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff92fc67288380 x1875449864245760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.621350:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864245760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.621369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.621378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574895.621381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574895.621389:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574895.621410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574895.621422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff92fc67288a80 x1875449864245888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.621434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864245888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.621443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.621446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574895.621450:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864245888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574895.621456:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574895.621474:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574895.621485:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff92fc67288700 x1875449864246016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574895.621497:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864246016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574895.621505:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574895.621509:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574895.621513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574896.068884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574896.132876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574897.092942:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574897.156850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574898.116877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574898.180910:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574899.140920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574899.204799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574900.164904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574900.228847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574900.740894:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574900.740905:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574900.740910:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574900.740921:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6651 00000100:00080000:2.0:1788574900.740932:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574900.740940:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574900.740978:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864246144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574900.741007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574900.741011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574900.741013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574900.741017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6651 00000100:00080000:2.0:1788574900.741022:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574900.741026:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574900.741037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574900.741052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864246272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574900.741060:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864246144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574900.741078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574900.741082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574900.741084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574900.741089:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6651 00000100:00080000:2.0:1788574900.741093:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574900.741096:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574900.741101:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574900.741114:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864246400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574900.741124:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574900.741129:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574900.741130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574900.741132:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574900.741139:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6651 00000100:00080000:2.0:1788574900.741144:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574900.741149:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574900.741155:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574900.741167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864246528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574900.741178:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff92fc4617c000 x1875449864246144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574900.741180:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574900.741188:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574900.741191:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574900.741200:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6651 00000100:00080000:2.0:1788574900.741206:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574900.741211:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864246144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574900.741212:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574900.741221:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574900.741229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574900.741248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788574900.741254:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864246656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574900.741258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864246144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574900.741290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574900.741317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574900.741330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff92fc4617e680 x1875449864246272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574900.741342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864246272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574900.741351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574900.741355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574900.741359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574900.741365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574900.741390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574900.741401:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff92fc4617ce00 x1875449864246400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574900.741410:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864246400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574900.741419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574900.741421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574900.741424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574900.741428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574900.741441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574900.741447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff92fc4617ea00 x1875449864246528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574900.741454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864246528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574900.741459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574900.741461:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574900.741463:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574900.741467:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574900.741476:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574900.741482:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff92fc4617d500 x1875449864246656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574900.741488:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864246656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574900.741493:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574900.741495:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574900.741498:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574901.188952:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574901.252903:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574902.212911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574902.276853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574903.236967:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574903.300911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574904.260896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574904.324820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574905.284954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574905.348875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574905.860848:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.860863:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574905.860869:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.860883:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6656 00000100:00080000:0.0:1788574905.860897:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.860907:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574905.860958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864246784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.860992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.860997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.860999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.861004:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6656 00000100:00080000:0.0:1788574905.861008:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.861013:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574905.861026:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.861044:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864246912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.861057:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.861062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574905.861064:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864246784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574905.861066:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.861076:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6656 00000100:00080000:0.0:1788574905.861081:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.861086:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574905.861092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.861107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864247040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.861119:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.861123:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.861125:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.861130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6656 00000100:00080000:0.0:1788574905.861134:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.861139:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574905.861144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574905.861149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574905.861167:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864247168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.861178:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.861183:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574905.861185:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.861189:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6656 00000100:00080000:0.0:1788574905.861194:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.861214:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574905.861220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff92fc67048e00 x1875449864246784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574905.861222:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574905.861243:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864247296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574905.861246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864246784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574905.861273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574905.861278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574905.861282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864246784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574905.861305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574905.861323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574905.861334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff92fc67049c00 x1875449864246912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574905.861347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864246912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574905.861357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574905.861360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574905.861364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864246912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574905.861371:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574905.861392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574905.861406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff92fc67049180 x1875449864247040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574905.861419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864247040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574905.861428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574905.861432:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574905.861436:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574905.861442:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574905.861464:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574905.861473:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff92fc67049500 x1875449864247168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574905.861482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864247168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574905.861487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574905.861490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574905.861492:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574905.861496:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574905.861508:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574905.861514:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff92fc6704bb80 x1875449864247296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574905.861521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864247296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574905.861527:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574905.861530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574905.861532:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574906.308846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574906.372931:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574907.332950:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574907.396883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574908.356888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574908.420888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574909.380897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574909.444896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574910.404875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574910.468899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574910.980859:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.980869:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574910.980874:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.980883:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6661 00000100:00080000:0.0:1788574910.980894:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.980901:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574910.980941:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864247424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.980967:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.980971:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.980973:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.980977:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6661 00000100:00080000:0.0:1788574910.980980:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.980983:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574910.980993:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574910.981005:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864247552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.981015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.981019:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574910.981021:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.981026:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6661 00000100:00080000:0.0:1788574910.981029:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.981032:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574910.981037:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574910.981047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864247680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574910.981058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864247424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574910.981062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.981076:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.981078:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.981084:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6661 00000100:00080000:0.0:1788574910.981088:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.981093:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574910.981098:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574910.981111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864247808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.981120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.981124:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574910.981126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.981130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6661 00000100:00080000:0.0:1788574910.981143:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.981147:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788574910.981154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574910.981175:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574910.981193:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864247936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574910.981245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff92fc6704b100 x1875449864247424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574910.981265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864247424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574910.981275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574910.981280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574910.981283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864247424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574910.981290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574910.981312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574910.981322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff92fc67049880 x1875449864247552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574910.981333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864247552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574910.981339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574910.981341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574910.981344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574910.981369:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574910.981382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574910.981389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff92fc6704ad80 x1875449864247680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574910.981397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864247680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574910.981402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574910.981404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574910.981406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574910.981410:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574910.981422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574910.981428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff92fc6704a680 x1875449864247808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574910.981435:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864247808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574910.981440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574910.981442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574910.981444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574910.981448:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574910.981468:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574910.981475:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff92fc6704aa00 x1875449864247936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574910.981484:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864247936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574910.981489:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574910.981491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574910.981493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864247936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574911.428883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574911.492851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574912.452870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574912.516852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574913.476988:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574913.540892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574914.500931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574914.564867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574915.524930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574915.588848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574916.100878:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574916.100887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574916.100891:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574916.100899:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6666 00000100:00080000:1.0:1788574916.100908:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574916.100914:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574916.100947:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864248064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574916.100973:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574916.100977:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574916.100979:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574916.100984:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6666 00000100:00080000:1.0:1788574916.100988:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574916.100994:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574916.101004:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574916.101013:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864248064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574916.101020:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864248192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574916.101028:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574916.101031:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574916.101032:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574916.101035:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6666 00000100:00080000:1.0:1788574916.101038:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574916.101041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574916.101044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574916.101053:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864248320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574916.101061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574916.101063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574916.101064:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574916.101066:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6666 00000100:00080000:1.0:1788574916.101069:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574916.101072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574916.101075:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574916.101079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574916.101089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864248448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574916.101095:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574916.101098:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574916.101099:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574916.101101:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6666 00000100:00080000:1.0:1788574916.101103:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574916.101105:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574916.101109:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574916.101115:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574916/real 1788574916] req@ffff92fc67289180 x1875449864248064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574916.101129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864248576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574916.101172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864248064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574916.101183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574916.101188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574916.101192:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864248064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574916.101217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574916.101241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574916.101252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574916/real 1788574916] req@ffff92fc67289500 x1875449864248192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574916.101261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864248192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574916.101272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574916.101274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574916.101276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574916.101280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574916.101292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574916.101301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574916/real 1788574916] req@ffff92fc6728bb80 x1875449864248320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574916.101313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864248320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574916.101321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574916.101323:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574916.101326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574916.101333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574916.101353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574916.101366:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574916/real 1788574916] req@ffff92fc67288700 x1875449864248448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574916.101378:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864248448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574916.101385:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574916.101388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574916.101393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574916.101399:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574916.101419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574916.101431:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574916/real 1788574916] req@ffff92fc67288a80 x1875449864248576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574916.101444:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864248576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574916.101452:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574916.101456:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574916.101459:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574916.548813:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574916.612801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574917.572903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574917.636836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574918.596889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574918.660797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574919.620913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574919.684859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574920.644902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574920.708788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574921.220841:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.220852:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574921.220857:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.220866:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6671 00000100:00080000:1.0:1788574921.220877:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.220883:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574921.220914:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864248704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.220936:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.220939:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574921.220941:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.220943:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6671 00000100:00080000:1.0:1788574921.220945:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.220947:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574921.220955:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574921.220964:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864248832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.220971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.220974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574921.220978:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574921.220979:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864248704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574921.220983:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6671 00000100:00080000:1.0:1788574921.220988:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.220990:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574921.220993:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574921.221002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864248960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.221008:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574921.221012:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574921.221013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.221015:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6671 00000100:00080000:1.0:1788574921.221018:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.221020:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574921.221023:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574921.221031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864249088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574921.221036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574921.221041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574921.221042:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574921.221046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574921.221049:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6671 00000100:00080000:1.0:1788574921.221053:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574921.221055:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574921.221058:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574921.221068:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864249216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574921.221108:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff92fc67288380 x1875449864248704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.221123:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864248704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.221131:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.221134:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574921.221137:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864248704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574921.221156:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574921.221174:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574921.221184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff92fc67289c00 x1875449864248832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.221192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864248832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.221198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.221199:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574921.221201:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574921.221205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574921.221216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574921.221223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff92fc6728a680 x1875449864248960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.221230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864248960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.221235:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.221237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574921.221239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864248960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574921.221242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574921.221252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574921.221258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff92fc6728b100 x1875449864249088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.221265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864249088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.221271:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.221273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574921.221275:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574921.221278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574921.221288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574921.221294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574921/real 1788574921] req@ffff92fc6728b800 x1875449864249216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574921.221300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864249216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574921.221305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574921.221307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574921.221309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788574921.668887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574921.732897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574922.692919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574922.756895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574923.716932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574923.780882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574924.740931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574924.804883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574925.764915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574925.828867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574926.340877:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574926.340887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574926.340891:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574926.340902:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6676 00000100:00080000:2.0:1788574926.340914:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574926.340922:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574926.340973:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864249344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574926.341001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574926.341006:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574926.341008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574926.341013:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6676 00000100:00080000:2.0:1788574926.341018:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574926.341023:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574926.341033:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574926.341061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864249472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574926.341069:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864249344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574926.341078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574926.341084:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574926.341086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574926.341091:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6676 00000100:00080000:2.0:1788574926.341096:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574926.341101:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574926.341106:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574926.341120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864249600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574926.341130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574926.341132:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574926.341134:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574926.341137:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6676 00000100:00080000:2.0:1788574926.341140:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574926.341143:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574926.341147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574926.341157:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864249728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574926.341163:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574926.341166:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574926.341167:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574926.341169:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6676 00000100:00080000:2.0:1788574926.341170:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574926.341173:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574926.341176:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574926.341184:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864249856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574926.341232:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574926.341262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574926.341381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574926.341410:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574926.788863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574926.852832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788574927.812864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574927.876860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574928.836848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574928.900843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788574929.413116:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788574929.413189:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574929.413211:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574929.413265:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574929.413280:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574929.413294:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788574929.413307:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574929.413332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff92fc4617d880 x1875449864249472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.413376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864249472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.413394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.413400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574929.413405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574929.413415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff92fc4617d500 x1875449864249600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.413431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864249600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.413439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.413443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574929.413447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574929.413455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff92fc4617ea00 x1875449864249728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.413468:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864249728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.413476:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.413479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574929.413483:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574929.413490:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff92fc4617ce00 x1875449864249856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.413501:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864249856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.413509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.413512:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574929.413515:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864249856:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574929.413531:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574926/real 1788574929] req@ffff92fc4617e300 x1875449864249344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574929.413544:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864249344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574929.413552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574929.413556:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574929.413559:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864249344:192.168.202.132@tcp:250:kworker.0 00010000:00010000:3.0:1788574929.861068:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574929.924859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574930.884947:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574930.948883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574931.460891:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574931.460903:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574931.460907:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574931.460919:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6682 00000100:00080000:2.0:1788574931.460931:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574931.460939:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574931.460992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864249984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574931.461045:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574931.461055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574931.461057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574931.461063:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6682 00000100:00080000:2.0:1788574931.461068:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574931.461074:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574931.461087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574931.461102:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864249984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574931.461107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864250112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574931.461118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574931.461121:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574931.461123:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574931.461127:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6682 00000100:00080000:2.0:1788574931.461131:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574931.461134:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574931.461140:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574931.461153:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864250240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574931.461164:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574931.461168:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574931.461169:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574931.461174:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6682 00000800:00000100:3.0:1788574931.461179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574931.461182:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574931.461189:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574931.461196:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574931.461208:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864250368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574931.461218:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574931.461221:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574931.461223:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574931.461227:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6682 00000100:00080000:2.0:1788574931.461232:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574931.461234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff92fc4617e680 x1875449864249984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574931.461241:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574931.461252:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574931.461262:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864249984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574931.461270:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864250496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574931.461275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.461283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574931.461290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864249984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574931.461326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574931.461360:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574931.461375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff92fc4617c000 x1875449864250112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.461389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864250112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.461398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.461401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574931.461404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574931.461411:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574931.461433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574931.461445:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff92fc4617fb80 x1875449864250240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.461456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864250240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.461464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.461467:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574931.461471:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574931.461476:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574931.461500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574931.461515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff92fc4617c380 x1875449864250368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.461527:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864250368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.461536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.461539:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574931.461542:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574931.461548:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574931.461568:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574931.461580:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574931/real 1788574931] req@ffff92fc4617d180 x1875449864250496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574931.461593:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864250496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574931.461603:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574931.461619:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574931.461623:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788574931.908965:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574931.972875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574932.932925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574932.996919:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788574933.956965:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574934.020840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574934.980892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574935.044831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574936.004913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574936.068814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574936.580868:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.580879:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574936.580884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.580896:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6687 00000100:00080000:1.0:1788574936.580906:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.580914:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574936.580978:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864250624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574936.580999:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.581003:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574936.581005:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.581010:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6687 00000100:00080000:1.0:1788574936.581014:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.581019:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574936.581026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864250624:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788574936.581031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574936.581048:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864250752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574936.581060:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.581064:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574936.581066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.581071:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6687 00000100:00080000:1.0:1788574936.581075:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.581077:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574936.581082:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574936.581091:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864250880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574936.581103:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788574936.581103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574936.581110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574936.581112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.581118:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6687 00000100:00080000:1.0:1788574936.581124:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.581128:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574936.581135:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574936.581150:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864251008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574936.581156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff92fc6728ad80 x1875449864250624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574936.581160:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574936.581167:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574936.581170:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574936.581177:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6687 00000100:00080000:1.0:1788574936.581182:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574936.581189:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574936.581196:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574936.581209:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864251136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574936.581209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864250624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.581238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.581244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574936.581249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864250624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574936.581274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574936.581300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574936.581314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff92fc67289880 x1875449864250752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.581327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864250752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.581336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.581339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574936.581342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574936.581349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574936.581370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574936.581381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff92fc67289f80 x1875449864250880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.581394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864250880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.581408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.581419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574936.581423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864250880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574936.581429:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574936.581451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574936.581464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff92fc67288000 x1875449864251008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.581476:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864251008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.581485:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.581488:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574936.581492:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574936.581498:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574936.581520:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574936.581531:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff92fc67288e00 x1875449864251136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574936.581544:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864251136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574936.581552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574936.581555:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574936.581559:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574937.028895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574937.092848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574938.052899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574938.116780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574939.076860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574939.140827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574940.100868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574940.164824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574941.124859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574941.188817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574941.700820:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.700828:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574941.700832:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.700840:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6692 00000100:00080000:1.0:1788574941.700848:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.700854:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574941.700888:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864251264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574941.700918:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.700933:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574941.700934:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.700937:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6692 00000100:00080000:1.0:1788574941.700940:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.700943:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574941.700952:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574941.700962:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864251392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574941.700969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864251264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574941.700984:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.700987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574941.700989:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.700994:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6692 00000100:00080000:1.0:1788574941.700998:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.701002:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574941.701007:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574941.701026:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864251520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574941.701035:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.701040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574941.701041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788574941.701043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574941.701050:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6692 00000100:00080000:1.0:1788574941.701054:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574941.701059:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574941.701065:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574941.701079:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864251648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574941.701087:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff92fc6728a300 x1875449864251264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574941.701094:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574941.701099:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574941.701101:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574941.701105:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864251264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574941.701108:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6692 00000100:00080000:1.0:1788574941.701114:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574941.701117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574941.701121:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:3.0:1788574941.701124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574941.701129:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864251264:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788574941.701130:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574941.701144:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864251776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574941.701150:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574941.701173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574941.701182:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff92fc6728b480 x1875449864251392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.701190:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864251392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.701196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.701199:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574941.701201:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574941.701204:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574941.701213:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574941.701220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff92fc6728aa00 x1875449864251520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.701226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864251520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.701231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.701233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574941.701235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574941.701238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574941.701247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574941.701252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff92fc67288e00 x1875449864251648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.701258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864251648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.701262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.701264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574941.701265:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574941.701281:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574941.701291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574941.701297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574941] req@ffff92fc67288000 x1875449864251776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574941.701304:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864251776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574941.701309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574941.701310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574941.701312:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864251776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574942.148934:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574942.212872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574943.172853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574943.236831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574944.196894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574944.260857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574945.220873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574945.284809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574946.244868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574946.308794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574946.820824:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.820835:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574946.820839:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.820849:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6697 00000100:00080000:2.0:1788574946.820860:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.820868:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574946.820904:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864251904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.820928:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.820932:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574946.820938:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.820940:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6697 00000100:00080000:2.0:1788574946.820944:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.820946:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574946.820954:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574946.820965:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864252032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.820972:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.820974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574946.820976:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.820978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6697 00000100:00080000:2.0:1788574946.820979:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.820985:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574946.820987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864251904:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788574946.820989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574946.820999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864252160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.821006:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.821008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574946.821009:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.821012:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6697 00000100:00080000:2.0:1788574946.821014:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.821016:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574946.821019:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574946.821027:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864252288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574946.821033:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574946.821036:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574946.821037:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574946.821039:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6697 00000100:00080000:2.0:1788574946.821041:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574946.821044:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574946.821046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574946.821053:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864252416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574946.821060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574946.821250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff92fc4617f100 x1875449864251904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.821267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864251904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.821276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.821278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574946.821281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864251904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574946.821287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574946.821310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574946.821325:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff92fc4617f800 x1875449864252032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.821338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864252032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.821348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.821352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574946.821355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574946.821377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574946.821392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574946.821400:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff92fc4617ca80 x1875449864252160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.821409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864252160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.821414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.821416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574946.821418:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574946.821422:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574946.821433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574946.821439:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff92fc4617c700 x1875449864252288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.821446:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864252288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.821451:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.821453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574946.821455:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574946.821458:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574946.821468:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574946.821480:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff92fc4617dc00 x1875449864252416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.821488:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864252416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574946.821493:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574946.821495:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574946.821497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574947.268856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574947.332880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574948.292899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574948.356894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574949.316874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574949.380893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574950.340902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574950.404883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574951.364962:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574951.428877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574951.940866:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.940877:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574951.940882:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.940893:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6702 00000100:00080000:2.0:1788574951.940904:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.940912:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574951.940954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864252544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.940979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.940982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574951.940988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.940990:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6702 00000100:00080000:2.0:1788574951.940992:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.940995:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574951.941005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574951.941014:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864252672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.941020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.941023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574951.941024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.941026:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6702 00000100:00080000:2.0:1788574951.941028:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.941030:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574951.941033:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574951.941035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864252544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574951.941043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864252800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.941054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.941057:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574951.941058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.941060:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6702 00000100:00080000:2.0:1788574951.941061:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.941063:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574951.941066:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574951.941074:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864252928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.941079:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574951.941081:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574951.941083:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574951.941085:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6702 00000100:00080000:2.0:1788574951.941096:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574951.941098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574951.941101:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574951.941108:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864253056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574951.941112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574951.941165:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff92fc4617f480 x1875449864252544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.941180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864252544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.941189:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.941192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574951.941195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864252544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574951.941201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574951.941220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574951.941232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff92fc4617ed80 x1875449864252672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.941244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864252672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.941269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.941273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574951.941276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574951.941303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574951.941323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574951.941352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff92fc4617dc00 x1875449864252800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.941363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864252800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.941371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.941374:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574951.941377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574951.941383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574951.941402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574951.941411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff92fc4617c700 x1875449864252928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.941421:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864252928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.941428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.941431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574951.941434:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864252928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574951.941441:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574951.941458:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574951.941467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff92fc4617ca80 x1875449864253056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.941479:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864253056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574951.941487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574951.941490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574951.941493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574952.388932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574952.452877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574953.412894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574953.476800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574954.436954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574954.500858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574955.460933:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574955.524884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574956.484844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574956.548839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574957.060858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574957.060866:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574957.060870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574957.060879:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6707 00000100:00080000:2.0:1788574957.060889:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574957.060895:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574957.060929:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864253184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574957.060952:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574957.060955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574957.060956:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574957.060959:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6707 00000100:00080000:2.0:1788574957.060961:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574957.060963:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574957.060972:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574957.060980:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864253312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574957.060988:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574957.060990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574957.060992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574957.060994:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6707 00000100:00080000:2.0:1788574957.060997:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574957.060999:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574957.061002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574957.061009:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864253184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788574957.061023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864253440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574957.061031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574957.061034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574957.061036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574957.061039:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6707 00000100:00080000:2.0:1788574957.061042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574957.061044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574957.061048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574957.061057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864253568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574957.061064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574957.061068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574957.061070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574957.061076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788574957.061077:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6707 00000100:00080000:2.0:1788574957.061086:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574957.061090:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574957.061097:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574957.061111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864253696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574957.061131:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574957/real 1788574957] req@ffff92fc4617f800 x1875449864253184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574957.061155:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864253184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574957.061166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574957.061171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574957.061175:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864253184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574957.061183:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574957.061207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574957.061218:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574957/real 1788574957] req@ffff92fc4617f100 x1875449864253312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574957.061226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864253312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574957.061233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574957.061235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574957.061237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574957.061256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574957.061269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574957.061276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574957/real 1788574957] req@ffff92fc4617d180 x1875449864253440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574957.061283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864253440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574957.061289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574957.061291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574957.061293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574957.061296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574957.061306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574957.061312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574957/real 1788574957] req@ffff92fc4617c380 x1875449864253568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574957.061319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864253568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574957.061324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574957.061326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574957.061328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574957.061331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574957.061341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574957.061347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574957/real 1788574957] req@ffff92fc4617fb80 x1875449864253696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574957.061353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864253696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574957.061358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574957.061360:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574957.061362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574957.508869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574957.572804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574958.532880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574958.596825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574959.556856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574959.620835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574960.580850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574960.644853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574961.604889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574961.668847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574962.180878:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574962.180894:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574962.180900:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574962.180920:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6712 00000100:00080000:0.0:1788574962.180934:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574962.180944:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574962.180998:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864253824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574962.181025:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574962.181028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574962.181029:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574962.181032:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6712 00000100:00080000:0.0:1788574962.181035:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574962.181037:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574962.181048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574962.181056:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864253952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574962.181064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574962.181067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574962.181068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574962.181071:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6712 00000100:00080000:0.0:1788574962.181075:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574962.181077:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864253824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574962.181078:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574962.181082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574962.181093:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864254080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574962.181099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574962.181102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574962.181103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574962.181105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6712 00000100:00080000:0.0:1788574962.181106:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574962.181108:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574962.181112:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574962.181118:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864254208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574962.181124:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574962.181125:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574962.181126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574962.181128:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6712 00000100:00080000:0.0:1788574962.181132:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574962.181134:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574962.181137:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574962.181143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864254336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574962.181166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574962.181232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff92fc67048000 x1875449864253824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.181260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864253824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.181269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.181275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574962.181278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864253824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788574962.181302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574962.181327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574962.181339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff92fc67049f80 x1875449864253952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.181348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864253952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.181357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.181362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574962.181366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864253952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574962.181372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788574962.181397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574962.181468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff92fc67048700 x1875449864254080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.181479:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864254080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.181487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.181490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574962.181494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574962.181500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574962.181524:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574962.181536:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff92fc67048380 x1875449864254208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.181545:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864254208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.181552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.181554:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574962.181669:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574962.181676:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788574962.181694:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788574962.181701:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574962/real 1788574962] req@ffff92fc6704a300 x1875449864254336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574962.181711:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864254336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574962.181716:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574962.181718:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574962.181801:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574962.628896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574962.692886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574963.652908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574963.716900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574964.676826:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574964.740853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574965.700878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574965.764829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574966.724846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574966.788847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574967.300851:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574967.300859:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574967.300862:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574967.300870:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6717 00000100:00080000:3.0:1788574967.300878:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574967.300884:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574967.300917:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864254464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574967.300945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574967.300950:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574967.300952:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574967.300956:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6717 00000100:00080000:3.0:1788574967.300960:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574967.300964:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574967.300974:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574967.300987:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864254592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574967.301021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574967.301028:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864254464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574967.301028:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574967.301031:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574967.301039:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6717 00000100:00080000:3.0:1788574967.301044:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574967.301048:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574967.301054:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574967.301068:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864254720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574967.301078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574967.301081:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574967.301083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574967.301087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6717 00000100:00080000:3.0:1788574967.301091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574967.301097:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574967.301100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788574967.301104:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574967.301118:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864254848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574967.301125:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574967.301128:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574967.301129:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574967.301132:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6717 00000100:00080000:3.0:1788574967.301135:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574967.301138:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574967.301142:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574967.301148:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff92fc67338380 x1875449864254464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574967.301152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864254976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574967.301190:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864254464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574967.301204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574967.301208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574967.301213:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864254464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574967.301237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574967.301263:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574967.301276:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff92fc67338a80 x1875449864254592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574967.301289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864254592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574967.301298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574967.301301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574967.301305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574967.301311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574967.301329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574967.301340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff92fc6733b800 x1875449864254720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574967.301352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864254720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574967.301360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574967.301364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574967.301367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574967.301383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574967.301404:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574967.301414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff92fc67339880 x1875449864254848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574967.301432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864254848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574967.301440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574967.301443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574967.301446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574967.301452:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574967.301469:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574967.301479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574967/real 1788574967] req@ffff92fc67339c00 x1875449864254976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574967.301499:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864254976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574967.301507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574967.301510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574967.301513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864254976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574967.748889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574967.812836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574968.772872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574968.836818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574969.796862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574969.860842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574970.820891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574970.884901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574971.844906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574971.908848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788574972.420853:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574972.420864:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574972.420869:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574972.420880:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6723 00000100:00080000:3.0:1788574972.420891:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574972.420899:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788574972.420937:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864255104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574972.420969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574972.420975:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574972.420977:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574972.420982:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6723 00000100:00080000:3.0:1788574972.420986:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574972.420990:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574972.421001:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574972.421014:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864255232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574972.421027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864255104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788574972.421027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574972.421036:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574972.421040:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574972.421044:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6723 00000100:00080000:3.0:1788574972.421048:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574972.421051:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574972.421055:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574972.421067:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864255360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574972.421074:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574972.421076:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574972.421078:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574972.421080:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6723 00000100:00080000:3.0:1788574972.421083:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574972.421088:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788574972.421090:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574972.421095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574972.421105:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864255488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574972.421111:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574972.421114:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574972.421115:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574972.421118:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6723 00000100:00080000:3.0:1788574972.421120:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574972.421123:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788574972.421126:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574972.421128:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff92fc67339180 x1875449864255104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574972.421135:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864255616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574972.421148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864255104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574972.421163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574972.421166:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574972.421169:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864255104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574972.421189:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574972.421203:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574972.421211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff92fc6733ad80 x1875449864255232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574972.421233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864255232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574972.421239:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574972.421241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574972.421243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574972.421247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574972.421262:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574972.421269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff92fc67339f80 x1875449864255360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574972.421276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864255360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574972.421281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574972.421283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574972.421285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574972.421289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574972.421299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574972.421305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff92fc67339c00 x1875449864255488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574972.421312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864255488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574972.421317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574972.421319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574972.421320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574972.421324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574972.421334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574972.421339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574972/real 1788574972] req@ffff92fc67339880 x1875449864255616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574972.421346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864255616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574972.421351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574972.421352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574972.421354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574972.868916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574972.932848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574973.892881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574973.956825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574974.916885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574974.980823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574975.940873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574976.004869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574976.964983:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574977.028799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574977.540849:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574977.540857:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574977.540861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574977.540869:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6728 00000100:00080000:0.0:1788574977.540878:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574977.540884:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574977.540916:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864255744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574977.540940:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574977.540945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574977.540946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574977.540951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6728 00000100:00080000:0.0:1788574977.540954:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574977.540958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574977.540969:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574977.540983:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864255872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574977.540994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574977.540997:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574977.540999:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574977.541002:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6728 00000100:00080000:0.0:1788574977.541006:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574977.541009:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574977.541014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574977.541022:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864255744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574977.541029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864256000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574977.541043:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574977.541047:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574977.541050:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574977.541055:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6728 00000100:00080000:0.0:1788574977.541069:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574977.541074:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574977.541078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574977.541082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574977.541096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864256128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574977.541103:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574977.541105:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574977.541107:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574977.541111:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6728 00000100:00080000:0.0:1788574977.541114:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574977.541118:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff92fc67048a80 x1875449864255744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574977.541119:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574977.541128:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574977.541139:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864255744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574977.541145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864256256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574977.541149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574977.541156:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574977.541163:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864255744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574977.541169:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574977.541187:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574977.541201:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff92fc6704a300 x1875449864255872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574977.541215:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864255872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574977.541225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574977.541229:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574977.541233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864255872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574977.541256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574977.541272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574977.541279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff92fc67048380 x1875449864256000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574977.541287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864256000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574977.541298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574977.541300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574977.541302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574977.541305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574977.541316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574977.541322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff92fc67048700 x1875449864256128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574977.541329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864256128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574977.541334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574977.541336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574977.541338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574977.541342:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574977.541351:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574977.541357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574977/real 1788574977] req@ffff92fc67049f80 x1875449864256256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574977.541364:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864256256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574977.541368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574977.541370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574977.541373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574977.988861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574978.052835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574979.012916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574979.076839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574980.036842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574980.100832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574981.060867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574981.124831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574982.084847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574982.148822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574982.660856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.660866:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574982.660871:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.660880:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6733 00000100:00080000:0.0:1788574982.660891:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.660899:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574982.660938:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864256384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574982.660966:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.660969:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574982.660970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.660973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6733 00000100:00080000:0.0:1788574982.660976:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.660979:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574982.660988:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574982.661002:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864256512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574982.661024:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864256384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574982.661028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.661035:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574982.661037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574982.661041:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6733 00000100:00080000:0.0:1788574982.661046:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.661050:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574982.661056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574982.661070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864256640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574982.661081:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.661083:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574982.661084:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574982.661089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788574982.661093:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6733 00000100:00080000:0.0:1788574982.661097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.661101:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574982.661105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574982.661115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864256768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574982.661122:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574982.661126:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574982.661128:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574982.661132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff92fc67048000 x1875449864256384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574982.661137:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6733 00000100:00080000:0.0:1788574982.661146:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574982.661152:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574982.661155:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864256384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574982.661160:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574982.661164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574982.661169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574982.661172:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864256384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788574982.661178:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864256896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574982.661195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574982.661213:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574982.661224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff92fc6704aa00 x1875449864256512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574982.661232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864256512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574982.661238:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574982.661240:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574982.661243:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574982.661246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574982.661256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574982.661263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff92fc6704a680 x1875449864256640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574982.661270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864256640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574982.661275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574982.661277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574982.661279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574982.661282:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574982.661292:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574982.661298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff92fc6704ad80 x1875449864256768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574982.661305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864256768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574982.661310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574982.661311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574982.661313:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574982.661316:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574982.661325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574982.661331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff92fc67049880 x1875449864256896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574982.661337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864256896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574982.661342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574982.661344:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574982.661346:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864256896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574983.108895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574983.172852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574984.132909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574984.196864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574985.156916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574985.220878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574986.180927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574986.244850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574987.204939:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574987.268847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788574987.780815:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574987.780826:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574987.780831:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574987.780841:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6738 00000100:00080000:2.0:1788574987.780853:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574987.780861:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788574987.780917:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864257024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574987.780946:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574987.780950:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574987.780952:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574987.780956:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6738 00000100:00080000:2.0:1788574987.780960:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574987.780965:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574987.780975:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574987.780989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864257152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574987.781000:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574987.781006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864257024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788574987.781006:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574987.781011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574987.781020:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6738 00000100:00080000:2.0:1788574987.781023:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574987.781026:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574987.781031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574987.781042:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864257280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574987.781051:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574987.781055:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574987.781057:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574987.781060:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6738 00000100:00080000:2.0:1788574987.781065:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574987.781069:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574987.781075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574987.781081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788574987.781088:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864257408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574987.781100:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574987.781104:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574987.781105:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574987.781109:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6738 00000100:00080000:2.0:1788574987.781113:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574987.781117:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788574987.781123:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574987.781129:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff92fc4617c000 x1875449864257024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574987.781137:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864257536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574987.781161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864257024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574987.781176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574987.781180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574987.781185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864257024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574987.781208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574987.781233:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574987.781246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff92fc4617e680 x1875449864257152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574987.781259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864257152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574987.781269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574987.781272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574987.781276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574987.781283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574987.781303:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574987.781314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff92fc4617e300 x1875449864257280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574987.781327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864257280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574987.781335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574987.781339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574987.781343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574987.781349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574987.781370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574987.781382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff92fc4617ce00 x1875449864257408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574987.781395:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864257408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574987.781403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574987.781406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574987.781410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574987.781416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574987.781446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574987.781458:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff92fc4617ea00 x1875449864257536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574987.781471:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864257536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574987.781480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574987.781483:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574987.781487:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574988.228945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574988.292859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574989.252878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574989.316833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574990.276926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574990.340833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574991.300941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574991.364906:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574992.324868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574992.388793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788574992.900875:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.900884:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574992.900889:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.900899:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6743 00000100:00080000:1.0:1788574992.900911:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.900918:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788574992.900957:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864257664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.900986:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.900990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574992.900992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.900995:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6743 00000100:00080000:1.0:1788574992.900999:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.901003:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574992.901014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574992.901026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864257792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.901044:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574992.901046:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864257664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788574992.901049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574992.901053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.901059:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6743 00000100:00080000:1.0:1788574992.901062:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.901066:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574992.901072:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574992.901084:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864257920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.901094:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.901098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574992.901100:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.901103:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6743 00000100:00080000:1.0:1788574992.901123:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.901127:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574992.901133:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574992.901145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864258048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574992.901155:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574992.901159:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574992.901161:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574992.901166:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6743 00000100:00080000:1.0:1788574992.901170:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574992.901174:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788574992.901180:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574992.901187:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574992.901194:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864258176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574992.901233:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574992.901260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574992.901291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574993.348857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574993.412841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574994.372896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574994.436786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574995.396861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574995.460856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788574995.973034:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788574995.973085:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574995.973102:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574995.973147:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574995.973158:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574995.973169:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788574995.973176:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788574995.973339:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff92fc6728b100 x1875449864258176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.973371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864258176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.973390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.973396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574995.973401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258176:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574995.973410:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff92fc67289880 x1875449864257792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.973424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864257792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.973434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.973438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574995.973441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574995.973447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff92fc6728ad80 x1875449864257920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.973457:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864257920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.973465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.973469:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574995.973472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864257920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788574995.973479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff92fc6728b800 x1875449864258048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.973492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864258048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.973499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.973502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574995.973505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788574995.973512:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574995] req@ffff92fc67289f80 x1875449864257664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574995.973524:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864257664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574995.973531:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574995.973534:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788574995.973537:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864257664:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788574996.420889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574996.484830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574997.444880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574997.508836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788574998.020893:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574998.020904:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574998.020908:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574998.020917:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6748 00000100:00080000:0.0:1788574998.020926:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574998.020933:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788574998.020961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864258304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574998.020992:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574998.020995:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574998.020997:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574998.021001:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6748 00000100:00080000:0.0:1788574998.021006:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574998.021011:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574998.021022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574998.021038:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864258432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574998.021048:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864258304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788574998.021052:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574998.021056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574998.021058:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574998.021064:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6748 00000100:00080000:0.0:1788574998.021071:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574998.021076:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574998.021082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574998.021096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864258560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574998.021108:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574998.021112:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574998.021115:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574998.021120:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6748 00000100:00080000:0.0:1788574998.021124:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574998.021129:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788574998.021136:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788574998.021137:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574998.021154:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864258688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574998.021165:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574998.021175:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574998.021177:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574998.021184:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6748 00000100:00100000:1.0:1788574998.021185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574998/real 1788574998] req@ffff92fc6704b100 x1875449864258304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574998.021188:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574998.021198:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788574998.021208:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574998.021217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864258304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574998.021226:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864258816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574998.021233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574998.021242:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574998.021247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864258304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788574998.021274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574998.021298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574998.021312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574998/real 1788574998] req@ffff92fc6704bb80 x1875449864258432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574998.021325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864258432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574998.021349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574998.021353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574998.021358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574998.021364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788574998.021391:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574998.021403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574998/real 1788574998] req@ffff92fc67049500 x1875449864258560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574998.021415:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864258560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574998.021423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574998.021426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574998.021430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788574998.021436:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574998.021454:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574998.021464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574998/real 1788574998] req@ffff92fc67049180 x1875449864258688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574998.021477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864258688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574998.021484:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574998.021488:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574998.021491:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788574998.021497:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788574998.021514:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788574998.021524:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574998/real 1788574998] req@ffff92fc67049c00 x1875449864258816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574998.021536:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864258816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574998.021544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574998.021546:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788574998.021550:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864258816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788574998.468896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574998.532838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788574999.492899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574999.556849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575000.516958:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575000.580894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575001.540868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575001.604854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575002.564937:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575002.628896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575003.140826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575003.140838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575003.140843:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575003.140854:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6753 00000100:00080000:2.0:1788575003.140866:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575003.140873:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575003.140919:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864258944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575003.140952:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575003.140957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575003.140959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575003.140963:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6753 00000100:00080000:2.0:1788575003.140967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575003.140972:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575003.140984:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575003.141000:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864259072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575003.141010:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575003.141012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575003.141014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575003.141017:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6753 00000100:00080000:2.0:1788575003.141021:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575003.141025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864258944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575003.141026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575003.141033:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575003.141045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864259200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575003.141058:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575003.141060:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575003.141062:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575003.141064:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6753 00000100:00080000:2.0:1788575003.141067:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575003.141069:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575003.141073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575003.141081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864259328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575003.141087:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575003.141090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575003.141092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575003.141094:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6753 00000100:00080000:2.0:1788575003.141097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575003.141099:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575003.141102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575003.141111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864259456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575003.141113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575003.141168:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575003/real 1788575003] req@ffff92fc4617d500 x1875449864258944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575003.141185:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864258944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575003.141195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575003.141198:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575003.141201:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864258944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575003.141229:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575003.141256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575003.141270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575003/real 1788575003] req@ffff92fc4617d880 x1875449864259072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575003.141283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864259072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575003.141294:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575003.141297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575003.141299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575003.141304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575003.141320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575003.141328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575003/real 1788575003] req@ffff92fc4617ea00 x1875449864259200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575003.141336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864259200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575003.141341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575003.141343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575003.141344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575003.141348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575003.141361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575003.141368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575003/real 1788575003] req@ffff92fc4617ce00 x1875449864259328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575003.141375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864259328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575003.141380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575003.141391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575003.141395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575003.141399:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575003.141412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575003.141419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575003/real 1788575003] req@ffff92fc4617e300 x1875449864259456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575003.141427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864259456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575003.141433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575003.141435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575003.141437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575003.588845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575003.652833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575004.612854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575004.676865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575005.636827:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575005.700829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575006.660870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575006.724854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575007.684877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575007.748881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575008.260911:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575008.260924:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575008.260929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575008.260941:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6758 00000100:00080000:3.0:1788575008.260952:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575008.260961:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575008.261008:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864259584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575008.261043:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575008.261060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575008.261062:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575008.261067:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6758 00000100:00080000:3.0:1788575008.261072:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575008.261078:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575008.261091:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575008.261107:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864259584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575008.261110:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864259712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575008.261130:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575008.261133:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575008.261136:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575008.261141:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6758 00000100:00080000:3.0:1788575008.261146:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575008.261151:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575008.261157:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575008.261173:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864259840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575008.261184:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575008.261188:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575008.261190:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575008.261194:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6758 00000100:00080000:3.0:1788575008.261199:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575008.261204:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575008.261210:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575008.261213:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575008.261227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864259968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575008.261243:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575008.261245:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575008.261247:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575008.261250:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6758 00000100:00080000:3.0:1788575008.261253:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575008.261256:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575008.261260:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575008.261272:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864260096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575008.261277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff92fc6733b800 x1875449864259584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575008.261302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864259584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575008.261312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575008.261316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575008.261319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864259584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575008.261344:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575008.261363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575008.261374:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff92fc67338a80 x1875449864259712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575008.261383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864259712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575008.261389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575008.261391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575008.261394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575008.261398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575008.261409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575008.261416:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff92fc67338380 x1875449864259840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575008.261424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864259840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575008.261429:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575008.261431:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575008.261433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575008.261437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575008.261449:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575008.261455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff92fc6733b480 x1875449864259968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575008.261463:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864259968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575008.261477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575008.261480:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575008.261482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864259968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575008.261486:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575008.261499:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575008.261505:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575008/real 1788575008] req@ffff92fc6733aa00 x1875449864260096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575008.261513:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864260096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575008.261519:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575008.261521:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575008.261523:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575008.708929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575008.772864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575009.732941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575009.796811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575010.756950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575010.820833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575011.780926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575011.844902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575012.804940:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575012.868870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575013.380937:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575013.380953:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575013.380959:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575013.380975:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6763 00000100:00080000:3.0:1788575013.380991:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575013.381001:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575013.381055:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864260224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575013.381092:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575013.381097:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575013.381099:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575013.381104:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6763 00000100:00080000:3.0:1788575013.381107:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575013.381111:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575013.381124:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575013.381139:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864260352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575013.381149:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575013.381154:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575013.381156:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575013.381157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864260224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575013.381162:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6763 00000100:00080000:3.0:1788575013.381166:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575013.381170:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575013.381173:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575013.381182:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864260480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575013.381189:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575013.381192:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575013.381193:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575013.381195:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6763 00000100:00080000:3.0:1788575013.381198:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575013.381200:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575013.381203:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575013.381219:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864260608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575013.381225:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575013.381228:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575013.381230:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575013.381233:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6763 00000100:00080000:3.0:1788575013.381236:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575013.381238:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575013.381242:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575013.381251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575013.381253:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864260736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575013.381313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff92fc6733bb80 x1875449864260224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575013.381345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864260224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575013.381359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575013.381364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575013.381369:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864260224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575013.381401:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575013.381436:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575013.381453:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff92fc67338000 x1875449864260352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575013.381467:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864260352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575013.381477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575013.381481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575013.381485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575013.381492:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575013.381514:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575013.381527:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff92fc67338700 x1875449864260480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575013.381540:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864260480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575013.381550:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575013.381554:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575013.381557:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575013.381564:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575013.381588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575013.381600:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff92fc67339500 x1875449864260608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575013.381614:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864260608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575013.381623:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575013.381627:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575013.381630:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575013.381637:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575013.381659:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575013.381671:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575013/real 1788575013] req@ffff92fc6733a300 x1875449864260736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575013.381684:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864260736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575013.381705:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575013.381709:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575013.381713:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575013.828912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575013.892824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575014.852864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575014.916916:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575015.876866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575015.940827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575016.900873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575016.964835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575017.924870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575017.988867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575018.500885:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575018.500897:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575018.500902:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575018.500914:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6769 00000100:00080000:1.0:1788575018.500927:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575018.500937:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575018.501005:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864260864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575018.501041:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575018.501046:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575018.501048:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575018.501053:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6769 00000100:00080000:1.0:1788575018.501057:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575018.501062:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575018.501075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575018.501090:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864260864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575018.501096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864260992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575018.501107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575018.501111:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575018.501113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575018.501118:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6769 00000100:00080000:1.0:1788575018.501123:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575018.501128:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575018.501135:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575018.501150:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864261120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575018.501162:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575018.501164:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575018.501166:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575018.501169:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6769 00000100:00080000:1.0:1788575018.501173:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575018.501175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575018.501180:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575018.501186:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575018.501196:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864261248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575018.501203:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575018.501206:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575018.501208:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575018.501210:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6769 00000100:00080000:1.0:1788575018.501212:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575018.501214:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575018.501217:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575018.501225:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864261376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575018.501244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff92fc6728a680 x1875449864260864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.501269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864260864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.501281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.501287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575018.501291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864260864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575018.501323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575018.501349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575018.501365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff92fc67289c00 x1875449864260992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.501379:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864260992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.501396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.501400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575018.501403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864260992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575018.501409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575018.501432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575018.501444:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff92fc67288380 x1875449864261120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.501456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864261120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.501465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.501468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575018.501470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575018.501474:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575018.501490:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575018.501498:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff92fc67288a80 x1875449864261248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.501506:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864261248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.501511:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.501513:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575018.501516:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575018.501520:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575018.501531:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575018.501537:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575018/real 1788575018] req@ffff92fc67288700 x1875449864261376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575018.501544:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864261376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575018.501549:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575018.501551:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575018.501553:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575018.948881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575019.012866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575019.972886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575020.036851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575020.996892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575021.060827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575022.020900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575022.084843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575023.044889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575023.108867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575023.620872:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575023.620883:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575023.620888:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575023.620900:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6774 00000100:00080000:1.0:1788575023.620912:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575023.620921:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575023.620971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864261504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575023.621005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575023.621011:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575023.621012:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575023.621016:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6774 00000100:00080000:1.0:1788575023.621019:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575023.621022:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575023.621033:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575023.621050:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864261632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575023.621059:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575023.621062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575023.621063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575023.621066:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6774 00000100:00080000:1.0:1788575023.621069:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575023.621071:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575023.621075:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575023.621081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864261760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575023.621090:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575023.621104:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575023.621105:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575023.621110:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6774 00000100:00080000:1.0:1788575023.621118:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575023.621121:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864261504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575023.621124:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575023.621136:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575023.621156:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864261888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575023.621167:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575023.621172:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575023.621174:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575023.621179:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6774 00000100:00080000:1.0:1788575023.621185:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575023.621190:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575023.621195:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575023.621207:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864262016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575023.621216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575023.621287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff92fc6728bb80 x1875449864261504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575023.621310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864261504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575023.621322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575023.621327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575023.621331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864261504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575023.621340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575023.621366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575023.621382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff92fc67289500 x1875449864261632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575023.621395:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864261632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575023.621414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575023.621419:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575023.621422:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575023.621450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575023.621474:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575023.621486:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff92fc67289180 x1875449864261760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575023.621499:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864261760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575023.621507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575023.621511:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575023.621514:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575023.621520:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575023.621540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575023.621552:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff92fc67288700 x1875449864261888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575023.621563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864261888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575023.621572:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575023.621575:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575023.621579:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864261888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575023.621592:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575023.621613:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575023.621625:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff92fc67288a80 x1875449864262016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575023.621637:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864262016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575023.621645:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575023.621648:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575023.621652:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575024.068982:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575024.132887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575025.092875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575025.156828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575026.116893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575026.180845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575027.140891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575027.204821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575028.164904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575028.228837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575028.740866:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.740877:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575028.740882:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.740892:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6779 00000100:00080000:3.0:1788575028.740902:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.740910:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575028.740943:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864262144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.740974:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.740979:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575028.740981:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.740986:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6779 00000100:00080000:3.0:1788575028.740990:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.740994:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575028.741003:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575028.741016:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864262144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575028.741017:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864262272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.741031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.741034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575028.741035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.741038:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6779 00000100:00080000:3.0:1788575028.741040:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.741042:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575028.741045:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575028.741052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864262400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.741059:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.741062:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575028.741064:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.741066:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6779 00000100:00080000:3.0:1788575028.741070:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.741072:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575028.741075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575028.741085:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864262528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575028.741089:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575028.741097:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.741100:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575028.741101:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.741105:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6779 00000100:00080000:3.0:1788575028.741109:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.741111:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575028.741114:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575028.741122:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864262656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575028.741132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff92fc67338e00 x1875449864262144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.741147:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864262144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.741156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.741160:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575028.741162:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864262144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575028.741180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575028.741195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575028.741204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff92fc6733a680 x1875449864262272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.741240:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864262272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.741246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.741249:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575028.741251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575028.741255:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575028.741270:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575028.741277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff92fc6733b100 x1875449864262400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.741285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864262400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.741290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.741292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575028.741294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575028.741297:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575028.741307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575028.741313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff92fc6733a300 x1875449864262528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.741325:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864262528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.741331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.741334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575028.741338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575028.741344:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575028.741363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575028.741375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff92fc67339500 x1875449864262656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575028.741389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864262656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575028.741398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575028.741402:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575028.741405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575029.188921:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575029.252842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575030.212877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575030.276851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575031.236855:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575031.300831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575032.260887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575032.324833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575033.284878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575033.348809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575033.860839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.860849:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575033.860855:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.860864:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6784 00000100:00080000:3.0:1788575033.860888:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.860898:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575033.860934:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864262784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.860969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.860974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.860976:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.860981:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6784 00000100:00080000:3.0:1788575033.860986:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.860990:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575033.861000:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.861028:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864262912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.861045:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.861048:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575033.861051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.861061:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6784 00000100:00100000:2.0:1788575033.861063:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864262784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575033.861067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.861083:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575033.861090:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.861111:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864263040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575033.861171:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575033.861207:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.861211:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.861214:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.861220:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6784 00000100:00080000:3.0:1788575033.861228:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.861233:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575033.861238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff92fc67338700 x1875449864262784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575033.861243:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575033.861269:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864263168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575033.861277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864262784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.861317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.861322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575033.861326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864262784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575033.861370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575033.861401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575033.861425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff92fc67338000 x1875449864262912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.861436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864262912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.861471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575033.861472:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575033.861481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575033.861488:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864262912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575033.861495:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575033.861514:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575033.861526:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff92fc6733bb80 x1875449864263040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575033.861530:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575033.861538:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575033.861550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864263040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575033.861551:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6784 00000100:00080000:3.0:1788575033.861564:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575033.861577:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575033.861580:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:2.0:1788575033.861584:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575033.861590:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263040:192.168.202.132@tcp:38:kworker.0 00000008:00100000:3.0:1788575033.861592:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575033.861603:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575033.861613:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864263296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575033.861626:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575033.861656:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff92fc6733aa00 x1875449864263168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.861670:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864263168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.861678:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.861682:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575033.861686:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575033.861697:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575033.861712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575033.861756:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff92fc6733b480 x1875449864263296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.861770:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864263296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.861776:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.861778:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575033.861781:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575034.308882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575034.372857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575035.332867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575035.396844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575036.356844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575036.420816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575037.380828:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575037.444820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575038.404825:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575038.468812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575038.980865:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575038.980873:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575038.980877:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575038.980887:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6789 00000100:00080000:3.0:1788575038.980896:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575038.980904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575038.980939:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864263424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575038.980968:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575038.980973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575038.980974:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575038.980978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6789 00000100:00080000:3.0:1788575038.980982:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575038.980986:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575038.980996:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575038.981010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864263424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575038.981011:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864263552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575038.981025:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575038.981028:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575038.981030:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575038.981034:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6789 00000100:00080000:3.0:1788575038.981037:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575038.981041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575038.981046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575038.981057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864263680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575038.981066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575038.981070:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575038.981073:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575038.981076:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575038.981081:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6789 00000100:00080000:3.0:1788575038.981085:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575038.981089:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575038.981094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575038.981105:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864263808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575038.981110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff92fc67338380 x1875449864263424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575038.981119:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575038.981127:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575038.981130:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575038.981134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6789 00000100:00080000:3.0:1788575038.981138:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575038.981141:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864263424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575038.981144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575038.981151:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575038.981161:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575038.981164:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864263936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575038.981169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575038.981177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864263424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575038.981205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575038.981230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575038.981242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff92fc67338a80 x1875449864263552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.981252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864263552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.981258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.981260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575038.981262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575038.981266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575038.981278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575038.981285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff92fc6733b800 x1875449864263680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.981292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864263680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.981297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.981299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575038.981301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575038.981304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575038.981313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575038.981318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff92fc67339880 x1875449864263808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.981324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864263808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.981328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.981330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575038.981331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575038.981334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575038.981342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575038.981352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff92fc67339c00 x1875449864263936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.981359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864263936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.981364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.981366:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575038.981367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864263936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575039.428840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575039.492823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575040.452820:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575040.516835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575041.476861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575041.540828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575042.500887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575042.564824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575043.524869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575043.588783:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575044.100810:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575044.100823:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575044.100828:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575044.100841:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6794 00000100:00080000:3.0:1788575044.100853:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575044.100862:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575044.100908:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864264064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575044.100936:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575044.100940:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575044.100942:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575044.100946:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6794 00000100:00080000:3.0:1788575044.100949:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575044.100953:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575044.100964:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575044.100977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864264192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575044.100988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864264064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575044.100999:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575044.101003:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575044.101004:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575044.101009:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6794 00000100:00080000:3.0:1788575044.101012:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575044.101015:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575044.101019:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575044.101038:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864264320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575044.101047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575044.101050:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575044.101051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575044.101055:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6794 00000100:00080000:3.0:1788575044.101058:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575044.101063:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575044.101067:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575044.101076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575044.101081:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864264448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575044.101091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575044.101094:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575044.101096:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575044.101100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6794 00000100:00080000:3.0:1788575044.101104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575044.101108:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575044.101113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575044.101125:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864264576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575044.101141:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575044/real 1788575044] req@ffff92fc67339f80 x1875449864264064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575044.101167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864264064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575044.101179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575044.101183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575044.101187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864264064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575044.101217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575044.101242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575044.101255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575044/real 1788575044] req@ffff92fc6733ad80 x1875449864264192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575044.101267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864264192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575044.101275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575044.101288:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575044.101291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575044.101298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575044.101319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575044.101330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575044/real 1788575044] req@ffff92fc67339180 x1875449864264320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575044.101341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864264320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575044.101349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575044.101352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575044.101355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575044.101361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575044.101380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575044.101390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575044/real 1788575044] req@ffff92fc67339c00 x1875449864264448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575044.101402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864264448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575044.101409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575044.101413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575044.101416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575044.101422:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575044.101441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575044.101452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575044/real 1788575044] req@ffff92fc67339880 x1875449864264576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575044.101464:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864264576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575044.101471:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575044.101474:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575044.101477:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575044.548886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575044.612839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575045.572888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575045.636871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575046.596867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575046.660847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575047.620877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575047.684856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575048.644860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575048.708815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575049.220841:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575049.220855:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575049.220858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575049.220866:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6799 00000100:00080000:1.0:1788575049.220874:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575049.220878:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575049.220906:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864264704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575049.220925:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575049.220932:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575049.220934:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575049.220937:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6799 00000100:00080000:1.0:1788575049.220939:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575049.220941:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575049.220948:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575049.220955:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864264832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575049.220962:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575049.220964:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575049.220965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575049.220967:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6799 00000100:00080000:1.0:1788575049.220969:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575049.220971:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575049.220974:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575049.220981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864264960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575049.220983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864264704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575049.220988:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575049.220993:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575049.220994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575049.220996:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6799 00000100:00080000:1.0:1788575049.220998:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575049.221000:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575049.221004:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575049.221009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864265088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575049.221014:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575049.221015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575049.221016:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575049.221019:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6799 00000100:00080000:1.0:1788575049.221020:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575049.221022:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575049.221025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575049.221031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864265216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575049.221061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575049.221101:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff92fc67288380 x1875449864264704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575049.221120:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864264704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575049.221131:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575049.221134:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575049.221138:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864264704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575049.221145:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575049.221165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575049.221176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff92fc67289c00 x1875449864264832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575049.221187:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864264832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575049.221201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575049.221204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575049.221207:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575049.221228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575049.221240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575049.221247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff92fc6728a680 x1875449864264960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575049.221253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864264960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575049.221258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575049.221260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575049.221262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864264960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575049.221265:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575049.221274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575049.221280:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff92fc67289f80 x1875449864265088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575049.221285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864265088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575049.221289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575049.221291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575049.221293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575049.221296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575049.221305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575049.221310:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575049/real 1788575049] req@ffff92fc6728b800 x1875449864265216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575049.221316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864265216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575049.221320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575049.221322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575049.221323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575049.668862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575049.732834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575050.692881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575050.756806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575051.716904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575051.780832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575052.740895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575052.804810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575053.764857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575053.828871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575054.340904:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575054.340914:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575054.340918:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575054.340929:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6804 00000100:00080000:2.0:1788575054.340940:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575054.340948:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575054.340986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864265344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575054.341017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575054.341021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575054.341023:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575054.341028:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6804 00000100:00080000:2.0:1788575054.341032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575054.341039:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575054.341049:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864265344:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788575054.341049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575054.341068:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864265472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575054.341082:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575054.341086:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575054.341088:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575054.341093:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6804 00000100:00080000:2.0:1788575054.341096:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575054.341100:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575054.341105:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575054.341118:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864265600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575054.341124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575054.341134:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575054.341139:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575054.341141:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575054.341147:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6804 00000100:00080000:2.0:1788575054.341151:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575054.341156:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575054.341161:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575054.341169:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff92fc4617e680 x1875449864265344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575054.341175:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864265728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575054.341190:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575054.341193:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864265344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575054.341196:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575054.341200:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575054.341209:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6804 00000100:00080000:2.0:1788575054.341215:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575054.341216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575054.341220:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:1.0:1788575054.341226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:2.0:1788575054.341227:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575054.341234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864265344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575054.341242:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864265856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575054.341270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575054.341299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575054.341314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff92fc4617c000 x1875449864265472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.341326:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864265472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.341335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.341338:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575054.341341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575054.341348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575054.341368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575054.341379:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff92fc4617fb80 x1875449864265600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.341389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864265600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.341397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.341400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575054.341403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575054.341409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575054.341429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575054.341440:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff92fc4617c380 x1875449864265728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.341460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864265728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.341468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.341471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575054.341474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575054.341480:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575054.341500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575054.341511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575054/real 1788575054] req@ffff92fc4617d180 x1875449864265856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575054.341524:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864265856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575054.341529:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575054.341532:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575054.341534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864265856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575054.788864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575054.852784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575055.812867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575055.876848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575056.836903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575056.900840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575057.860864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575057.924821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575058.884924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575058.948890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575059.460812:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.460823:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575059.460828:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.460839:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6810 00000100:00080000:2.0:1788575059.460850:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.460860:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575059.460910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864265984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.460945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.460948:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575059.460949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.460952:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6810 00000100:00080000:2.0:1788575059.460955:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.460958:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575059.460970:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575059.460983:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864266112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.460995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.460999:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575059.461001:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.461006:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6810 00000100:00080000:2.0:1788575059.461009:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.461012:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575059.461016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575059.461024:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864266240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.461033:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.461037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575059.461041:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575059.461045:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864265984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575059.461045:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6810 00000100:00080000:2.0:1788575059.461050:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.461056:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575059.461064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575059.461072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864266368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575059.461079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575059.461082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575059.461084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575059.461089:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6810 00000100:00080000:2.0:1788575059.461095:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575059.461100:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575059.461106:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575059.461119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864266496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575059.461177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575059.461224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575059.461248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575059.461269:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575059.908849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575059.972887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575060.932924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575060.996883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575061.956831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575062.020893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788575062.533099:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788575062.533168:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575062.533191:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575062.533244:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575062.533255:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575062.533265:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575062.533276:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788575062.533343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff92fc4617ca80 x1875449864266240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575062.533376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864266240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575062.533394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575062.533402:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575062.533408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575062.533418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff92fc4617c700 x1875449864266368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575062.533433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864266368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575062.533442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575062.533446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575062.533451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575062.533461:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff92fc4617dc00 x1875449864266496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575062.533485:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864266496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575062.533496:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575062.533500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575062.533505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266496:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575062.533513:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff92fc4617f100 x1875449864265984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575062.533529:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864265984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575062.533540:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575062.533545:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575062.533549:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864265984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575062.533560:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575059/real 1788575062] req@ffff92fc4617f800 x1875449864266112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575062.533577:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864266112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575062.533587:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575062.533592:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575062.533596:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266112:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788575062.980984:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575063.044876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575064.004923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575064.068862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575064.580850:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.580861:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575064.580865:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.580878:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6815 00000100:00080000:2.0:1788575064.580891:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.580899:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575064.580947:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864266624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.580985:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.580989:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575064.580990:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.580994:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6815 00000100:00080000:2.0:1788575064.580997:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.581000:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575064.581012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575064.581025:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864266752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.581045:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.581053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575064.581054:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.581059:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6815 00000100:00100000:1.0:1788575064.581060:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864266624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575064.581063:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.581072:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575064.581080:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575064.581097:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864266880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.581115:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.581119:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575064.581121:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.581127:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6815 00000100:00080000:2.0:1788575064.581131:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.581138:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575064.581145:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575064.581154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575064.581162:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864267008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575064.581175:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575064.581179:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575064.581182:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575064.581186:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6815 00000100:00080000:2.0:1788575064.581191:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575064.581195:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575064.581202:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575064.581214:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff92fc4617f800 x1875449864266624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575064.581219:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864267136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575064.581294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864266624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575064.581313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575064.581319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575064.581324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864266624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575064.581356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575064.581398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575064.581413:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff92fc4617f100 x1875449864266752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575064.581426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864266752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575064.581436:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575064.581440:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575064.581443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575064.581450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575064.581473:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575064.581485:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff92fc4617dc00 x1875449864266880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575064.581497:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864266880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575064.581505:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575064.581508:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575064.581511:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864266880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575064.581519:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575064.581539:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575064.581570:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff92fc4617c700 x1875449864267008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575064.581583:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864267008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575064.581591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575064.581594:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575064.581597:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575064.581603:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575064.581625:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575064.581637:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff92fc4617ca80 x1875449864267136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575064.581648:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864267136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575064.581662:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575064.581665:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575064.581668:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575065.028977:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575065.092905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575066.052957:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575066.116864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575067.076938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575067.140887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575068.100991:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575068.164851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575069.124944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575069.188887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575069.700885:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575069.700897:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575069.700902:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575069.700913:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6820 00000100:00080000:2.0:1788575069.700926:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575069.700934:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575069.700981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864267264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575069.701016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575069.701020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575069.701021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575069.701024:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6820 00000100:00080000:2.0:1788575069.701029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575069.701034:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575069.701044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575069.701057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864267392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575069.701066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575069.701072:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575069.701074:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864267264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575069.701075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575069.701081:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6820 00000100:00080000:2.0:1788575069.701089:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575069.701093:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575069.701096:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575069.701107:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864267520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575069.701114:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575069.701118:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575069.701120:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575069.701126:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6820 00000100:00080000:2.0:1788575069.701130:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575069.701134:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575069.701139:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575069.701150:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864267648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575069.701173:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575069.701176:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575069.701178:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575069.701182:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6820 00000100:00080000:2.0:1788575069.701185:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575069.701188:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575069.701193:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575069.701198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575069.701211:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864267776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575069.701247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff92fc4617ed80 x1875449864267264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575069.701266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864267264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575069.701275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575069.701278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575069.701281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864267264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575069.701302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575069.701327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575069.701336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff92fc4617f480 x1875449864267392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575069.701344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864267392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575069.701351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575069.701353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575069.701355:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575069.701360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575069.701372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575069.701379:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff92fc4617ca80 x1875449864267520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575069.701386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864267520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575069.701391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575069.701393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575069.701395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575069.701399:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575069.701410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575069.701415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff92fc4617c700 x1875449864267648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575069.701423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864267648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575069.701428:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575069.701430:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575069.701431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575069.701435:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575069.701446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575069.701456:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff92fc4617dc00 x1875449864267776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575069.701469:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864267776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575069.701477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575069.701480:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575069.701484:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864267776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575070.148991:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575070.212808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575071.172941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575071.236911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575072.196917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575072.260807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575073.220858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575073.284853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575074.244870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575074.308812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575074.820852:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.820860:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575074.820864:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.820871:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6825 00000100:00080000:0.0:1788575074.820880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.820887:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575074.820919:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864267904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.820953:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.820960:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575074.820966:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.820970:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6825 00000100:00080000:0.0:1788575074.820975:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.820979:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575074.820989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575074.821004:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864268032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.821012:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.821015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575074.821016:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.821019:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6825 00000100:00080000:0.0:1788575074.821022:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.821025:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575074.821029:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575074.821034:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864267904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575074.821039:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864268160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.821049:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.821051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575074.821052:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.821054:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6825 00000100:00080000:0.0:1788575074.821056:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.821059:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575074.821063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575074.821071:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864268288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575074.821078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575074.821080:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575074.821081:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575074.821084:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6825 00000100:00080000:0.0:1788575074.821093:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575074.821096:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575074.821099:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575074.821105:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575074.821108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864268416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575074.821151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff92fc67048e00 x1875449864267904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.821172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864267904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.821193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.821197:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575074.821202:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864267904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575074.821211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575074.821234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575074.821271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff92fc67049c00 x1875449864268032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.821285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864268032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.821302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.821306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575074.821309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575074.821335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575074.821356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575074.821368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff92fc67049180 x1875449864268160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.821389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864268160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.821400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.821404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575074.821407:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575074.821414:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575074.821436:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575074.821447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff92fc67049500 x1875449864268288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.821461:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864268288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.821469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.821473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575074.821477:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575074.821484:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575074.821504:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575074.821515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575074] req@ffff92fc6704bb80 x1875449864268416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575074.821533:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864268416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575074.821539:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575074.821541:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575074.821543:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575075.268912:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575075.332823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575076.292904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575076.356828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575077.316854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575077.380838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575078.340895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575078.404810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575079.364835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575079.428812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575079.940926:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575079.940945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575079.940948:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575079.940956:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6830 00000100:00080000:3.0:1788575079.940965:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575079.940972:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575079.941006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864268544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575079.941036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575079.941041:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575079.941043:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575079.941047:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6830 00000100:00080000:3.0:1788575079.941051:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575079.941056:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575079.941067:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575079.941081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864268672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575079.941090:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575079.941095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575079.941096:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575079.941104:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6830 00000100:00080000:3.0:1788575079.941108:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575079.941109:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864268544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575079.941113:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575079.941120:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575079.941130:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864268800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575079.941136:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575079.941138:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575079.941139:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575079.941142:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6830 00000100:00080000:3.0:1788575079.941143:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575079.941146:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575079.941149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575079.941155:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864268928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575079.941162:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575079.941164:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575079.941165:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575079.941168:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6830 00000100:00080000:3.0:1788575079.941170:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575079.941173:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575079.941177:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575079.941183:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864269056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575079.941210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575079.941273:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff92fc6733b800 x1875449864268544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.941296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864268544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.941310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.941315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575079.941319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864268544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575079.941364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575079.941388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575079.941402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff92fc67338a80 x1875449864268672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.941413:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864268672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.941422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.941426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575079.941430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575079.941437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575079.941456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575079.941466:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff92fc67338380 x1875449864268800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.941478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864268800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.941495:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.941499:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575079.941503:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575079.941509:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575079.941529:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575079.941541:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff92fc6733b480 x1875449864268928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.941553:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864268928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.941562:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.941566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575079.941569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864268928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575079.941575:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575079.941594:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575079.941604:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff92fc6733aa00 x1875449864269056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575079.941615:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864269056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575079.941623:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575079.941626:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575079.941629:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575080.388926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575080.452878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575081.412926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575081.476859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575082.436875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575082.500847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575083.460935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575083.524842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575084.484884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575084.548833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575085.060843:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575085.060855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575085.060859:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575085.060869:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6835 00000100:00080000:3.0:1788575085.060879:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575085.060885:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575085.060918:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864269184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575085.060944:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575085.060947:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575085.060949:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575085.060951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6835 00000100:00080000:3.0:1788575085.060953:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575085.060956:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575085.060963:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575085.060974:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864269312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575085.060988:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575085.060991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575085.060992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575085.060998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864269184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575085.060998:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6835 00000100:00080000:3.0:1788575085.061004:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575085.061010:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575085.061014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575085.061022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864269440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575085.061029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575085.061032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575085.061034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575085.061038:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6835 00000100:00080000:3.0:1788575085.061042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575085.061046:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575085.061052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575085.061066:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575085.061068:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864269568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575085.061080:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575085.061085:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575085.061086:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575085.061091:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6835 00000100:00080000:3.0:1788575085.061095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575085.061100:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575085.061105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575085.061107:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575085/real 1788575085] req@ffff92fc6733bb80 x1875449864269184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575085.061122:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864269696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575085.061125:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864269184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575085.061151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575085.061159:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575085.061162:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864269184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575085.061188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575085.061211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575085.061225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575085/real 1788575085] req@ffff92fc67338000 x1875449864269312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575085.061237:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864269312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575085.061244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575085.061246:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575085.061248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575085.061253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575085.061268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575085.061275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575085/real 1788575085] req@ffff92fc67338700 x1875449864269440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575085.061283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864269440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575085.061288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575085.061290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575085.061293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575085.061299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575085.061319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575085.061332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575085/real 1788575085] req@ffff92fc67339500 x1875449864269568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575085.061359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864269568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575085.061368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575085.061372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575085.061375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575085.061382:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575085.061405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575085.061439:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575085/real 1788575085] req@ffff92fc6733a300 x1875449864269696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575085.061452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864269696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575085.061460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575085.061464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575085.061467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575085.508954:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575085.572847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575086.532925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575086.596895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575087.556941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575087.620819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575088.580871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575088.644838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575089.604928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575089.668862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575090.180879:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.180890:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575090.180894:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.180905:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6840 00000100:00080000:0.0:1788575090.180917:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.180924:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575090.180974:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864269824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575090.181003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.181007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575090.181009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.181013:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6840 00000100:00080000:0.0:1788575090.181017:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.181021:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575090.181031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575090.181045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864269952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575090.181061:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864269824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575090.181068:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.181072:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575090.181074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.181080:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6840 00000100:00080000:0.0:1788575090.181083:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.181088:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575090.181093:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575090.181107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864270080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575090.181118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.181122:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788575090.181126:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575090.181127:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575090.181135:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6840 00000100:00080000:0.0:1788575090.181141:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.181146:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575090.181153:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575090.181165:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff92fc6704b100 x1875449864269824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575090.181172:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864270208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575090.181183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864269824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575090.181187:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575090.181198:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575090.181202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575090.181203:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575090.181210:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788575090.181214:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6840 00000100:00100000:1.0:1788575090.181218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864269824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575090.181223:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575090.181231:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575090.181237:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575090.181242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575090.181257:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864270336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575090.181264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575090.181278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff92fc67049880 x1875449864269952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.181288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864269952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.181295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.181297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575090.181301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864269952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575090.181306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575090.181317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575090.181323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff92fc6704ad80 x1875449864270080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.181330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864270080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.181340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.181342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575090.181344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575090.181347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575090.181357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575090.181363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff92fc6704a680 x1875449864270208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.181370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864270208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.181375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.181377:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575090.181379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575090.181388:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575090.181398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575090.181404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575090/real 1788575090] req@ffff92fc6704aa00 x1875449864270336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575090.181411:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864270336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575090.181415:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575090.181417:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575090.181419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575090.628874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575090.692852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575091.652903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575091.716796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575092.676870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575092.740823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575093.700950:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575093.764808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575094.724943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575094.788851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575095.300897:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575095.300909:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575095.300914:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575095.300927:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6845 00000100:00080000:2.0:1788575095.300941:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575095.300950:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575095.301008:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864270464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575095.301036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575095.301040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575095.301041:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575095.301044:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6845 00000100:00080000:2.0:1788575095.301047:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575095.301050:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575095.301059:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575095.301070:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864270592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575095.301077:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575095.301079:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575095.301080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575095.301082:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6845 00000100:00080000:2.0:1788575095.301084:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575095.301086:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575095.301089:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575095.301096:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864270464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575095.301097:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864270720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575095.301110:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575095.301112:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575095.301113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575095.301115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6845 00000100:00080000:2.0:1788575095.301118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575095.301119:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575095.301122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575095.301130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864270848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575095.301135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575095.301137:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575095.301138:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575095.301140:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6845 00000100:00080000:2.0:1788575095.301141:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575095.301144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575095.301146:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575095.301153:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864270976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575095.301197:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575095.301252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff92fc4617f100 x1875449864270464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.301275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864270464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.301286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.301290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575095.301294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864270464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575095.301301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575095.301322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575095.301335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff92fc4617f800 x1875449864270592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.301347:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864270592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.301355:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.301358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575095.301361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575095.301391:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575095.301412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575095.301423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff92fc4617d180 x1875449864270720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.301435:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864270720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.301443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.301446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575095.301449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575095.301455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575095.301474:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575095.301485:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff92fc4617c380 x1875449864270848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.301496:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864270848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.301504:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.301507:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575095.301511:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575095.301516:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575095.301533:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575095.301543:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575095/real 1788575095] req@ffff92fc4617fb80 x1875449864270976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575095.301560:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864270976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575095.301569:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575095.301572:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575095.301575:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864270976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575095.748981:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575095.812878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575096.772886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575096.836828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575097.796891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575097.860836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575098.820812:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575098.884823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575099.844892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575099.908843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575100.420803:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.420811:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575100.420814:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.420822:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6851 00000100:00080000:3.0:1788575100.420830:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.420843:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575100.420877:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864271104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.420894:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.420897:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575100.420898:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.420900:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6851 00000100:00080000:3.0:1788575100.420903:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.420905:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575100.420911:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575100.420919:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864271232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.420926:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.420928:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575100.420929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.420933:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6851 00000100:00100000:1.0:1788575100.420934:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864271104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575100.420935:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.420940:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575100.420945:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575100.420951:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864271360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.420957:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.420959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575100.420960:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.420963:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6851 00000100:00080000:3.0:1788575100.420964:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.420966:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575100.420969:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575100.420976:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864271488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575100.420981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575100.420983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575100.420984:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575100.420986:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6851 00000800:00000100:1.0:1788575100.420990:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575100.420991:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575100.420995:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575100.420998:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575100.421005:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864271616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575100.421026:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff92fc6733b100 x1875449864271104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.421038:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864271104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.421045:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.421047:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575100.421050:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864271104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575100.421066:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575100.421084:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575100.421092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff92fc6733a680 x1875449864271232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.421103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864271232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.421109:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.421112:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575100.421115:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575100.421120:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575100.421138:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575100.421148:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff92fc67338e00 x1875449864271360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.421158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864271360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.421165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.421168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575100.421171:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575100.421176:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575100.421194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575100.421202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff92fc6733a300 x1875449864271488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.421208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864271488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.421217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.421219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575100.421221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575100.421224:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575100.421234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575100.421239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575100/real 1788575100] req@ffff92fc67339500 x1875449864271616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575100.421245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864271616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575100.421249:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575100.421251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575100.421252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575100.868809:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575100.932823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575101.892982:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575101.956878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575102.916927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575102.980884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575103.940971:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575104.004883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575104.964867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575105.028851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575105.540863:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575105.540872:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575105.540877:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575105.540887:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6856 00000100:00080000:2.0:1788575105.540898:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575105.540907:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575105.540941:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864271744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575105.540969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575105.540973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575105.540974:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575105.540977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6856 00000100:00080000:2.0:1788575105.540980:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575105.540983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575105.540991:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575105.541002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864271872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575105.541009:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864271744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575105.541011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575105.541017:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575105.541019:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575105.541022:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6856 00000100:00080000:2.0:1788575105.541025:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575105.541028:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575105.541031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575105.541040:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864272000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575105.541047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575105.541049:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575105.541051:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575105.541053:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6856 00000100:00080000:2.0:1788575105.541062:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575105.541064:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575105.541069:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575105.541075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575105.541077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864272128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575105.541083:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575105.541085:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575105.541086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575105.541089:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6856 00000100:00080000:2.0:1788575105.541091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575105.541093:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575105.541096:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575105.541102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864272256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575105.541123:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff92fc4617c000 x1875449864271744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575105.541195:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864271744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575105.541204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575105.541207:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575105.541211:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864271744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575105.541232:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575105.541349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575105.541405:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff92fc4617e680 x1875449864271872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575105.541421:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864271872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575105.541430:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575105.541434:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575105.541437:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864271872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575105.541442:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575105.541462:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575105.541473:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff92fc4617e300 x1875449864272000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575105.541481:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864272000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575105.541490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575105.541492:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575105.541494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575105.541498:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575105.541510:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575105.541524:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff92fc4617ce00 x1875449864272128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575105.541530:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864272128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575105.541535:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575105.541537:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575105.541539:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575105.541542:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575105.541551:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575105.541557:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575105/real 1788575105] req@ffff92fc4617ea00 x1875449864272256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575105.541563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864272256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575105.541568:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575105.541569:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575105.541571:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575105.988919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575106.052851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575107.012875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575107.076827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575108.036914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575108.100831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575109.060890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575109.124855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575110.084892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575110.148835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575110.660875:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.660884:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575110.660888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.660898:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6861 00000100:00080000:3.0:1788575110.660906:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.660912:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575110.660941:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864272384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.660965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.660968:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575110.660969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.660971:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6861 00000100:00080000:3.0:1788575110.660973:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.660975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575110.660982:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575110.660989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864272512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.660998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.661001:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575110.661003:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.661006:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6861 00000100:00080000:3.0:1788575110.661009:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.661014:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575110.661020:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575110.661021:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864272384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575110.661032:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864272640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.661041:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.661043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575110.661045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.661047:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6861 00000100:00080000:3.0:1788575110.661050:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.661052:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575110.661055:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575110.661063:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864272768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.661068:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.661071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575110.661072:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.661073:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6861 00000100:00080000:3.0:1788575110.661075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.661080:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575110.661081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788575110.661083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575110.661090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864272896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575110.661117:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff92fc67338700 x1875449864272384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.661129:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864272384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.661135:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.661138:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575110.661141:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864272384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575110.661146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575110.661165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575110.661176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff92fc67338000 x1875449864272512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.661188:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864272512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.661196:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.661198:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575110.661200:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575110.661217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575110.661230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575110.661237:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff92fc6733bb80 x1875449864272640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.661243:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864272640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.661248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.661250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575110.661252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575110.661255:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575110.661264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575110.661269:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff92fc6733aa00 x1875449864272768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.661275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864272768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.661280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.661281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575110.661283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575110.661285:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575110.661294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575110.661298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff92fc6733b480 x1875449864272896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575110.661304:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864272896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575110.661308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575110.661310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575110.661311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864272896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575111.108909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575111.172794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575112.132888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575112.196838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575113.156919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575113.220834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575114.180920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575114.244830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575115.204854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575115.268814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575115.780836:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575115.780846:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575115.780850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575115.780859:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6866 00000100:00080000:2.0:1788575115.780869:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575115.780885:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575115.780918:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864273024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575115.780969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575115.780973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575115.780974:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575115.780978:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6866 00000100:00080000:2.0:1788575115.780982:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575115.780986:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575115.780994:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575115.781006:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864273152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575115.781014:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864273024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575115.781019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575115.781023:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575115.781024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575115.781028:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6866 00000100:00080000:2.0:1788575115.781032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575115.781036:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575115.781040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575115.781052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864273280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575115.781060:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575115.781063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575115.781065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575115.781067:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6866 00000100:00080000:2.0:1788575115.781071:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575115.781074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575115.781078:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575115.781084:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575115.781095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864273408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575115.781107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575115.781117:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff92fc4617d880 x1875449864273024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575115.781118:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575115.781122:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575115.781132:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6866 00000100:00080000:2.0:1788575115.781139:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575115.781144:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575115.781149:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864273024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575115.781149:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575115.781160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.781164:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575115.781164:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864273536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575115.781170:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864273024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575115.781193:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575115.781210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575115.781218:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff92fc4617d500 x1875449864273152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.781225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864273152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.781231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.781233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575115.781235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575115.781238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575115.781256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575115.781262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff92fc4617ea00 x1875449864273280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.781269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864273280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.781277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.781279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575115.781281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575115.781284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575115.781293:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575115.781298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff92fc4617ce00 x1875449864273408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.781304:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864273408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.781308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.781311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575115.781314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575115.781319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575115.781343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575115.781353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff92fc4617e300 x1875449864273536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575115.781363:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864273536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575115.781368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575115.781370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575115.781372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575116.228859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575116.292826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575117.252854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575117.316814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575118.276832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575118.340833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575119.300873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575119.364842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575120.324822:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575120.388837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575120.900847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575120.900857:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575120.900865:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575120.900874:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6871 00000100:00080000:3.0:1788575120.900884:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575120.900892:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575120.900938:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864273664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575120.900963:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575120.900967:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575120.900969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575120.900977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6871 00000100:00080000:3.0:1788575120.900981:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575120.900985:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575120.900995:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575120.901009:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864273792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575120.901016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575120.901029:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575120.901032:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575120.901034:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864273664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575120.901037:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6871 00000100:00080000:3.0:1788575120.901042:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575120.901046:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575120.901050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575120.901065:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864273920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575120.901072:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575120.901074:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575120.901075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575120.901078:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6871 00000100:00080000:3.0:1788575120.901080:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575120.901082:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575120.901085:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575120.901092:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864274048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575120.901101:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575120.901101:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575120.901106:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575120.901107:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575120.901110:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6871 00000100:00080000:3.0:1788575120.901112:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575120.901114:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575120.901117:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575120.901123:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864274176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575120.901150:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff92fc67338380 x1875449864273664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.901168:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864273664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.901179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.901183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575120.901186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864273664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575120.901207:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575120.901230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575120.901242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff92fc67338a80 x1875449864273792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.901254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864273792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.901262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.901265:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575120.901268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575120.901273:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575120.901287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575120.901305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff92fc6733b800 x1875449864273920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.901316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864273920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.901324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.901327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575120.901330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864273920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575120.901335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575120.901352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575120.901363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff92fc67339880 x1875449864274048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.901375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864274048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.901382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.901386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575120.901389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575120.901394:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575120.901411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575120.901420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff92fc67339c00 x1875449864274176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575120.901431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864274176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575120.901438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575120.901441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575120.901444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575121.348874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575121.412837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575122.372865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575122.436858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575123.396858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575123.460834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575124.420840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575124.484820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575125.444871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575125.508838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575126.020830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575126.020840:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575126.020844:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575126.020852:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6876 00000100:00080000:2.0:1788575126.020868:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575126.020874:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575126.020909:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864274304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575126.020941:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575126.020943:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575126.020945:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575126.020948:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6876 00000100:00080000:2.0:1788575126.020951:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575126.020954:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575126.020963:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575126.020977:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864274304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575126.020977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864274432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575126.020990:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575126.020993:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575126.020994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575126.020997:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6876 00000100:00080000:2.0:1788575126.020999:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575126.021003:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575126.021008:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575126.021019:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864274560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575126.021029:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575126.021033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575126.021035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575126.021039:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6876 00000100:00080000:2.0:1788575126.021043:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575126.021046:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575126.021051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575126.021060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864274688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575126.021066:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274432:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788575126.021071:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575126.021075:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575126.021076:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575126.021079:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6876 00000100:00080000:2.0:1788575126.021082:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575126.021085:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575126.021089:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575126.021097:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575126.021102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864274816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575126.021138:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575126.021153:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575126.468888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575126.532814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575127.492848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575127.556833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575128.516892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575128.580825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788575129.093191:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788575129.093243:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575129.093258:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575129.093295:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575129.093304:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575129.093312:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575129.093318:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575129.093424:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575126/real 1788575129] req@ffff92fc4617c380 x1875449864274688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575129.093469:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864274688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575129.093483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575129.093489:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575129.093494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575129.093504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575126/real 1788575129] req@ffff92fc4617d180 x1875449864274816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575129.093517:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864274816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575129.093532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575129.093536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575129.093539:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274816:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575129.093547:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575126/real 1788575129] req@ffff92fc4617c000 x1875449864274432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575129.093560:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864274432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575129.093568:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575129.093571:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575129.093575:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575129.093582:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575126/real 1788575129] req@ffff92fc4617fb80 x1875449864274560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575129.093596:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864274560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575129.093618:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575129.093623:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575129.093626:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864274560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575129.093634:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575126/real 1788575129] req@ffff92fc4617e680 x1875449864274304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575129.093647:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864274304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575129.093660:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575129.093664:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575129.093666:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864274304:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575129.540865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575129.604814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575130.564886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575130.628850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575131.140836:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575131.140844:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575131.140848:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575131.140856:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6881 00000100:00080000:2.0:1788575131.140864:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575131.140871:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575131.140904:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864274944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575131.140944:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575131.140948:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575131.140951:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575131.140955:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6881 00000100:00080000:2.0:1788575131.140959:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575131.140962:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575131.140971:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575131.140982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864275072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575131.140991:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575131.140994:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575131.140995:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575131.140999:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6881 00000100:00080000:2.0:1788575131.141001:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575131.141005:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575131.141009:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575131.141019:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864275200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575131.141027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575131.141030:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575131.141031:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575131.141039:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6881 00000100:00080000:2.0:1788575131.141042:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575131.141045:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575131.141050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575131.141059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864275328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575131.141067:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575131.141070:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575131.141071:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575131.141074:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6881 00000100:00080000:2.0:1788575131.141077:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575131.141081:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575131.141084:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575131.141094:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864275456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575131.141287:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864274944:192.168.202.132@tcp:250:kworker.0 00000800:00000100:1.0:1788575131.141346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575131.141384:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575131/real 1788575131] req@ffff92fc4617f800 x1875449864274944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575131.141399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864274944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575131.141408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575131.141412:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575131.141415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864274944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575131.141421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575131.141441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575131.141452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575131/real 1788575131] req@ffff92fc4617f100 x1875449864275072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575131.141463:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864275072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575131.141470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575131.141473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575131.141475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575131.141481:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575131.141496:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575131.141513:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575131/real 1788575131] req@ffff92fc4617dc00 x1875449864275200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575131.141522:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864275200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575131.141528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575131.141531:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575131.141534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575131.141538:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575131.141556:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575131.141564:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575131/real 1788575131] req@ffff92fc4617c700 x1875449864275328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575131.141573:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864275328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575131.141579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575131.141582:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575131.141584:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575131.141589:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575131.141603:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575131.141611:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575131/real 1788575131] req@ffff92fc4617ca80 x1875449864275456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575131.141620:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864275456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575131.141626:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575131.141628:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575131.141631:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575131.588862:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575131.652810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575132.612818:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575132.676765:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575133.636865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575133.700794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575134.660869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575134.724847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575135.684900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575135.748886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575136.260847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575136.260857:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575136.260860:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575136.260870:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6886 00000100:00080000:2.0:1788575136.260880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575136.260888:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575136.260925:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864275584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575136.260951:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575136.260955:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575136.260957:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575136.260960:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6886 00000100:00080000:2.0:1788575136.260964:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575136.260968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575136.260978:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575136.260993:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864275712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575136.261004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575136.261008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575136.261010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575136.261014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6886 00000100:00080000:2.0:1788575136.261018:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575136.261025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864275584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575136.261026:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575136.261043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575136.261068:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864275840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575136.261080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575136.261085:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575136.261087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575136.261091:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6886 00000100:00080000:2.0:1788575136.261096:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575136.261100:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575136.261107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575136.261112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575136.261131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864275968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575136.261143:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575136.261147:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575136.261149:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575136.261153:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6886 00000100:00080000:2.0:1788575136.261158:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575136.261166:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff92fc4617f480 x1875449864275584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575136.261167:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575136.261180:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575136.261192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864275584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575136.261198:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864276096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575136.261204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575136.261215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575136.261220:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864275584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575136.261230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575136.261248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575136.261263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff92fc4617ed80 x1875449864275712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575136.261296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864275712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575136.261302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575136.261304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575136.261306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575136.261331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575136.261346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575136.261353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff92fc4617ca80 x1875449864275840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575136.261360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864275840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575136.261364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575136.261366:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575136.261368:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575136.261372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575136.261387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575136.261393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff92fc4617c700 x1875449864275968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575136.261400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864275968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575136.261408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575136.261410:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575136.261416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864275968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575136.261419:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575136.261430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575136.261436:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575136/real 1788575136] req@ffff92fc4617dc00 x1875449864276096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575136.261442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864276096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575136.261447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575136.261449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575136.261451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575136.708883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575136.772823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575137.732872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575137.796847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575138.756908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575138.821067:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575139.780806:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575139.844883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575140.804875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575140.868836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575141.380793:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.380801:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575141.380804:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.380813:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6891 00000100:00080000:3.0:1788575141.380823:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.380830:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575141.380858:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864276224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.380890:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.380893:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575141.380896:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.380900:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6891 00000100:00080000:3.0:1788575141.380905:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.380909:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575141.380919:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575141.380924:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864276224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575141.380937:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864276352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.380949:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.380953:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575141.380955:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.380959:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6891 00000100:00080000:3.0:1788575141.380987:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.380992:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575141.380997:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788575141.381002:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575141.381010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864276480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.381020:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.381023:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575141.381026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.381030:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6891 00000100:00080000:3.0:1788575141.381034:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.381037:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575141.381042:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575141.381051:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff92fc67339180 x1875449864276224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575141.381056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864276608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575141.381075:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575141.381078:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575141.381079:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575141.381082:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6891 00000100:00100000:2.0:1788575141.381086:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864276224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575141.381087:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575141.381093:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575141.381098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575141.381106:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864276736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575141.381112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575141.381119:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575141.381124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864276224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575141.381153:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575141.381172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575141.381188:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff92fc6733ad80 x1875449864276352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575141.381196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864276352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575141.381202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575141.381204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575141.381206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575141.381210:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575141.381220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575141.381226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff92fc67339f80 x1875449864276480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575141.381233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864276480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575141.381237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575141.381239:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575141.381241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575141.381245:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575141.381255:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575141.381260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff92fc67339c00 x1875449864276608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575141.381266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864276608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575141.381270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575141.381272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575141.381274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575141.381278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575141.381289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575141.381294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575141/real 1788575141] req@ffff92fc67339880 x1875449864276736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575141.381301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864276736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575141.381305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575141.381307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575141.381310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575141.828817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575141.892824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575142.852835:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575142.916837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575143.876868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575143.940831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575144.900844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575144.964801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575145.924888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575145.988842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575146.500829:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.500835:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575146.500839:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.500850:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6897 00000100:00080000:3.0:1788575146.500858:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.500863:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575146.500891:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864276864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.500911:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.500914:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575146.500915:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.500918:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6897 00000100:00080000:3.0:1788575146.500920:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.500922:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575146.500931:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575146.500938:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864276992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.500945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.500948:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575146.500949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.500951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6897 00000100:00080000:3.0:1788575146.500953:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.500960:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575146.500962:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864276864:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788575146.500965:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575146.500991:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864277120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.501001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.501005:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575146.501007:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.501010:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6897 00000100:00080000:3.0:1788575146.501032:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.501038:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575146.501044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575146.501053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575146.501059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864277248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575146.501073:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575146.501076:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575146.501077:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575146.501083:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6897 00000100:00080000:3.0:1788575146.501087:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575146.501090:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575146.501094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575146.501108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864277376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575146.501114:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff92fc6733b800 x1875449864276864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575146.501154:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864276864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575146.501166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575146.501170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575146.501173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864276864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575146.501201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575146.501227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575146.501239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff92fc67338a80 x1875449864276992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575146.501247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864276992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575146.501253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575146.501255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575146.501258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864276992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575146.501262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575146.501275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575146.501281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff92fc67338380 x1875449864277120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575146.501288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864277120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575146.501292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575146.501294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575146.501296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575146.501299:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575146.501309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575146.501315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff92fc6733b480 x1875449864277248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575146.501321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864277248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575146.501326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575146.501328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575146.501330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575146.501333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575146.501343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575146.501348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575146/real 1788575146] req@ffff92fc6733aa00 x1875449864277376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575146.501354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864277376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575146.501366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575146.501368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575146.501370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575146.948838:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575147.012823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575147.972852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575148.036794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575148.996908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575149.060823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575150.020870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575150.084827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575151.044866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575151.108829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575151.620805:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.620814:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575151.620817:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.620825:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6902 00000100:00080000:2.0:1788575151.620834:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.620840:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575151.620871:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864277504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.620896:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.620900:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575151.620905:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.620907:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6902 00000100:00080000:2.0:1788575151.620910:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.620912:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575151.620921:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575151.620930:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864277632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.620941:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.620945:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575151.620947:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.620953:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6902 00000100:00100000:1.0:1788575151.620954:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864277504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575151.620959:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.620965:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575151.620971:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575151.620986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864277760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.620994:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.620997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575151.620998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.621002:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6902 00000100:00080000:2.0:1788575151.621007:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.621011:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575151.621020:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575151.621028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575151.621034:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864277888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.621047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.621051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575151.621053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.621056:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6902 00000100:00080000:2.0:1788575151.621060:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.621074:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575151.621075:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff92fc4617f100 x1875449864277504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575151.621079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575151.621094:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864278016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575151.621100:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864277504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.621113:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.621117:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575151.621119:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864277504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575151.621126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575151.621146:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575151.621155:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff92fc4617f800 x1875449864277632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.621163:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864277632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.621169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.621172:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575151.621174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575151.621193:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575151.621207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575151.621213:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff92fc4617e680 x1875449864277760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.621221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864277760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.621232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.621234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575151.621237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575151.621241:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575151.621252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575151.621262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff92fc4617fb80 x1875449864277888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.621269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864277888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.621275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.621277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575151.621279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864277888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575151.621283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575151.621294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575151.621300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff92fc4617c000 x1875449864278016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.621312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864278016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.621317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.621319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575151.621321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575152.068850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575152.132792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575153.092868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575153.156830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575154.116878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575154.180794:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575155.140865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575155.204826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575156.164904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575156.228852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575156.740851:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.740860:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575156.740864:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.740872:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6907 00000100:00080000:0.0:1788575156.740881:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.740888:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575156.740922:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864278144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575156.740945:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.740948:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575156.740950:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.740952:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6907 00000100:00080000:0.0:1788575156.740955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.740957:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575156.740965:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575156.740973:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864278272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575156.740982:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.740984:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575156.740988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575156.740990:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864278144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575156.740992:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6907 00000100:00080000:0.0:1788575156.740996:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.741001:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575156.741005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575156.741016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864278400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575156.741026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.741030:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575156.741047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.741052:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6907 00000100:00080000:0.0:1788575156.741062:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.741066:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575156.741072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788575156.741074:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575156.741090:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864278528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575156.741099:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575156.741101:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575156.741102:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575156.741105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6907 00000100:00080000:0.0:1788575156.741108:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575156.741112:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575156.741116:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575156.741126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864278656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575156.741126:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff92fc67048000 x1875449864278144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.741163:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864278144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.741173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.741177:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575156.741180:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864278144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575156.741205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575156.741225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575156.741235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff92fc67049f80 x1875449864278272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.741244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864278272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.741250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.741252:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575156.741254:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575156.741257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575156.741269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575156.741275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff92fc67048700 x1875449864278400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.741283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864278400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.741288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.741289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575156.741291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575156.741294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575156.741305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575156.741311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff92fc67048380 x1875449864278528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.741318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864278528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.741328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.741330:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575156.741333:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575156.741336:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575156.741350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575156.741360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff92fc6704a300 x1875449864278656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.741372:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864278656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.741381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.741385:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575156.741388:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575157.188839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575157.252827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575158.212896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575158.276818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575159.236821:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575159.300826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575160.260875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575160.324859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575161.284858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575161.348857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575161.860856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575161.860867:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575161.860871:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575161.860888:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6912 00000100:00080000:3.0:1788575161.860897:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575161.860903:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575161.860933:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864278784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575161.860956:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575161.860961:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575161.860962:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575161.860965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6912 00000100:00080000:3.0:1788575161.860967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575161.860970:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575161.860978:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575161.860989:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864278912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575161.860998:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575161.861002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864278784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575161.861002:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575161.861005:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575161.861012:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6912 00000100:00080000:3.0:1788575161.861015:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575161.861017:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575161.861020:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575161.861029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864279040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575161.861036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575161.861038:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575161.861039:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575161.861042:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6912 00000100:00080000:3.0:1788575161.861044:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575161.861047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575161.861052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575161.861059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864279168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575161.861065:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575161.861068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575161.861069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575161.861071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6912 00000100:00080000:3.0:1788575161.861075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575161.861076:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575161.861080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575161.861084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575161.861092:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864279296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575161.861118:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff92fc6733bb80 x1875449864278784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575161.861134:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864278784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575161.861142:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575161.861145:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575161.861148:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864278784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575161.861166:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575161.861182:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575161.861190:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff92fc67338000 x1875449864278912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575161.861198:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864278912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575161.861204:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575161.861206:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575161.861208:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864278912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575161.861212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575161.861224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575161.861234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff92fc67338700 x1875449864279040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575161.861245:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864279040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575161.861254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575161.861257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575161.861261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575161.861268:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575161.861288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575161.861296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff92fc67339500 x1875449864279168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575161.861308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864279168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575161.861316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575161.861319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575161.861323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575161.861330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575161.861351:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575161.861361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff92fc6733a300 x1875449864279296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575161.861368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864279296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575161.861374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575161.861376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575161.861378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575162.308871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575162.372837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575163.332876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575163.396807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575164.356879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575164.420846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575165.380864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575165.444799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575166.404850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575166.468840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575166.980886:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575166.980896:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575166.980901:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575166.980914:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6917 00000100:00080000:3.0:1788575166.980928:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575166.980938:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575166.980986:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864279424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575166.981021:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575166.981026:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575166.981029:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575166.981034:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6917 00000100:00080000:3.0:1788575166.981039:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575166.981044:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575166.981056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575166.981068:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864279552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575166.981076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575166.981079:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575166.981081:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575166.981084:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6917 00000100:00080000:3.0:1788575166.981087:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575166.981089:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575166.981092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575166.981099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864279680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575166.981106:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575166.981109:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575166.981111:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575166.981118:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6917 00000100:00100000:1.0:1788575166.981120:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864279424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575166.981121:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575166.981126:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575166.981132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575166.981143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864279808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575166.981161:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575166.981163:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575166.981165:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575166.981168:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6917 00000100:00080000:3.0:1788575166.981171:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575166.981173:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575166.981177:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575166.981186:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864279936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575166.981209:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575166.981261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff92fc67338e00 x1875449864279424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.981282:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864279424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.981295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.981300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575166.981304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864279424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575166.981314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575166.981340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575166.981354:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff92fc6733a680 x1875449864279552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.981368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864279552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.981377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.981381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575166.981384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575166.981426:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575166.981451:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575166.981464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff92fc6733b100 x1875449864279680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.981478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864279680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.981486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.981490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575166.981493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575166.981500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575166.981522:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575166.981534:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff92fc6733a300 x1875449864279808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.981548:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864279808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.981556:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.981561:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575166.981564:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575166.981571:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575166.981592:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575166.981604:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff92fc67339500 x1875449864279936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575166.981615:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864279936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575166.981630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575166.981633:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575166.981637:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864279936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575167.428878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575167.492822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575168.452837:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575168.516822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575169.476854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575169.540823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575170.500842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575170.564830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575171.524890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575171.588814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575172.100817:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575172.100825:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575172.100828:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575172.100837:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6922 00000100:00080000:0.0:1788575172.100858:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575172.100867:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575172.100904:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864280064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575172.100937:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575172.100940:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575172.100942:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575172.100945:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6922 00000100:00080000:0.0:1788575172.100948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575172.100951:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575172.100959:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575172.100971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864280192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575172.100979:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575172.100982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575172.100984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575172.100988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864280064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575172.100989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6922 00000100:00080000:0.0:1788575172.100994:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575172.100998:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575172.101002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575172.101010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864280320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575172.101016:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575172.101018:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575172.101020:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575172.101022:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6922 00000100:00080000:0.0:1788575172.101039:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575172.101042:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575172.101046:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575172.101050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575172.101058:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864280448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575172.101065:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575172.101068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575172.101069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575172.101071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6922 00000100:00080000:0.0:1788575172.101074:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575172.101076:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575172.101079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575172.101087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864280576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575172.101091:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575172/real 1788575172] req@ffff92fc67048a80 x1875449864280064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575172.101124:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864280064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575172.101132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575172.101136:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575172.101139:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864280064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575172.101158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575172.101175:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575172.101184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575172/real 1788575172] req@ffff92fc6704a300 x1875449864280192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575172.101192:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864280192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575172.101198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575172.101201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575172.101203:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575172.101206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575172.101216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575172.101222:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575172/real 1788575172] req@ffff92fc67048380 x1875449864280320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575172.101229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864280320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575172.101235:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575172.101237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575172.101239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575172.101242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575172.101252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575172.101258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575172/real 1788575172] req@ffff92fc67048700 x1875449864280448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575172.101264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864280448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575172.101269:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575172.101271:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575172.101274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575172.101277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575172.101287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575172.101292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575172/real 1788575172] req@ffff92fc67049f80 x1875449864280576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575172.101299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864280576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575172.101304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575172.101305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575172.101307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575172.548847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575172.612851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575173.572867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575173.636797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575174.596829:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575174.660802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575175.620929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575175.684871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575176.644853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575176.708857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575177.220863:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575177.220875:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575177.220881:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575177.220893:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6927 00000100:00080000:3.0:1788575177.220905:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575177.220913:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575177.220955:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864280704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575177.220986:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575177.220990:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575177.220991:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575177.220996:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6927 00000100:00080000:3.0:1788575177.221000:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575177.221003:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575177.221017:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575177.221026:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864280704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575177.221037:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864280832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575177.221055:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575177.221058:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575177.221060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575177.221065:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6927 00000100:00080000:3.0:1788575177.221072:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575177.221075:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575177.221080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575177.221092:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864280960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575177.221101:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575177.221119:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575177.221123:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575177.221125:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575177.221131:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6927 00000100:00080000:3.0:1788575177.221136:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575177.221141:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575177.221147:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff92fc67338700 x1875449864280704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575177.221151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575177.221176:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864280704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.221187:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.221190:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575177.221194:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864280704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575177.221221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575177.221225:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864281088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575177.221244:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575177.221247:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575177.221249:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575177.221253:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6927 00000100:00080000:3.0:1788575177.221258:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575177.221261:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788575177.221266:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788575177.221270:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575177.221280:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff92fc67338000 x1875449864280832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575177.221285:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864281216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575177.221298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864280832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.221304:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.221306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575177.221308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575177.221313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575177.221325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575177.221331:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff92fc6733bb80 x1875449864280960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.221339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864280960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.221343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.221345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575177.221347:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864280960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575177.221357:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575177.221368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575177.221377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff92fc6733aa00 x1875449864281088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.221384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864281088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.221392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.221395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575177.221398:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575177.221479:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575177.221514:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575177.221536:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575177/real 1788575177] req@ffff92fc6733b480 x1875449864281216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575177.221563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864281216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575177.221571:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575177.221575:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575177.221579:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575177.668902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575177.732846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575178.692851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575178.756835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575179.716851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575179.780845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575180.740790:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575180.804850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575181.764839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575181.828802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575182.340828:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575182.340835:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575182.340839:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575182.340846:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6932 00000100:00080000:3.0:1788575182.340854:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575182.340859:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575182.340887:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864281344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575182.340909:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575182.340912:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575182.340913:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575182.340916:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6932 00000100:00080000:3.0:1788575182.340918:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575182.340919:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575182.340927:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575182.340936:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864281472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575182.340946:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575182.340949:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575182.340950:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575182.340954:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6932 00000100:00100000:0.0:1788575182.340955:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864281344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575182.340956:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575182.340961:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575182.340963:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575182.340971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864281600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575182.340981:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575182.340983:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575182.340985:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575182.340989:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6932 00000100:00080000:3.0:1788575182.340992:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575182.340996:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575182.341001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575182.341012:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864281728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575182.341021:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575182.341024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575182.341026:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575182.341029:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6932 00000800:00000100:0.0:1788575182.341035:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575182.341035:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575182.341050:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575182.341056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575182.341068:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864281856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575182.341077:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff92fc67338380 x1875449864281344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575182.341092:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864281344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575182.341103:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575182.341107:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575182.341111:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864281344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575182.341133:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575182.341151:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575182.341160:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff92fc67338a80 x1875449864281472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575182.341167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864281472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575182.341173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575182.341176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575182.341179:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575182.341184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575182.341200:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575182.341211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff92fc6733b800 x1875449864281600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575182.341222:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864281600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575182.341228:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575182.341230:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575182.341232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575182.341235:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575182.341248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575182.341254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff92fc67339880 x1875449864281728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575182.341261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864281728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575182.341265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575182.341267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575182.341269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575182.341272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575182.341280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575182.341285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575182/real 1788575182] req@ffff92fc67339c00 x1875449864281856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575182.341291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864281856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575182.341300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575182.341301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575182.341303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864281856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575182.788846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575182.852790:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575183.812844:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575183.876813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575184.836840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575184.900784:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575185.860843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575185.924798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575186.884841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575186.948855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575187.460853:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575187.460862:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575187.460866:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575187.460875:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6938 00000100:00080000:2.0:1788575187.460884:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575187.460889:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575187.460923:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864281984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575187.460954:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575187.460957:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575187.460959:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575187.460961:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6938 00000100:00080000:2.0:1788575187.460964:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575187.460967:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575187.460975:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575187.460984:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864282112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575187.460991:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575187.460994:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575187.460995:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575187.460998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864281984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575187.461000:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6938 00000100:00080000:2.0:1788575187.461002:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575187.461006:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575187.461011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575187.461018:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864282240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575187.461024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575187.461026:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575187.461028:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575187.461030:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6938 00000100:00080000:2.0:1788575187.461032:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575187.461034:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575187.461038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575187.461043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864282368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575187.461054:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575187.461056:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575187.461059:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575187.461060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575187.461063:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6938 00000100:00080000:2.0:1788575187.461066:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575187.461074:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575187.461077:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575187.461084:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864282496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575187.461091:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff92fc4617d180 x1875449864281984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575187.461109:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864281984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575187.461120:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575187.461124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575187.461128:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864281984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575187.461148:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575187.461165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575187.461187:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff92fc4617c380 x1875449864282112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575187.461196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864282112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575187.461202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575187.461203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575187.461206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575187.461209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575187.461224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575187.461233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff92fc4617e300 x1875449864282240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575187.461244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864282240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575187.461251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575187.461255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575187.461258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575187.461264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575187.461280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575187.461287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff92fc4617ce00 x1875449864282368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575187.461294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864282368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575187.461298:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575187.461300:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575187.461303:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575187.461306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575187.461317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575187.461322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575187/real 1788575187] req@ffff92fc4617ea00 x1875449864282496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575187.461328:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864282496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575187.461333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575187.461336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575187.461339:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575187.908848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575187.972811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575188.932887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575188.996786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575189.956903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575190.020900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575190.980919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575191.044774:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575192.004831:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575192.068833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575192.580835:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575192.580845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575192.580848:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575192.580856:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6943 00000100:00080000:2.0:1788575192.580884:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575192.580891:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575192.580920:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864282624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575192.580940:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575192.580943:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575192.580944:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575192.580947:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6943 00000100:00080000:2.0:1788575192.580949:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575192.580950:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575192.580958:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575192.580965:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864282752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575192.580971:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575192.580973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575192.580975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575192.580977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6943 00000100:00080000:2.0:1788575192.580978:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575192.580980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575192.580983:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575192.580990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864282880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575192.580995:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864282624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575192.580998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575192.581001:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575192.581003:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575192.581006:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6943 00000100:00080000:2.0:1788575192.581009:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575192.581012:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575192.581016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575192.581028:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864283008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575192.581037:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575192.581040:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575192.581042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575192.581046:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6943 00000100:00080000:2.0:1788575192.581050:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575192.581054:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575192.581058:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575192.581070:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864283136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575192.581081:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575192.581108:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575192.581121:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575192.581134:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575193.028867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575193.092860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575194.052843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575194.116828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575195.076879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575195.140820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788575195.652999:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:3.0:1788575195.653039:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575195.653054:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575195.653105:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575195.653114:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575195.653121:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575195.653129:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788575195.653287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff92fc4617ea00 x1875449864282880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.653312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864282880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.653338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.653343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575195.653347:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575195.653355:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff92fc4617ce00 x1875449864283008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.653369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864283008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.653376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.653380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575195.653383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575195.653389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff92fc4617e300 x1875449864283136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.653401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864283136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.653408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.653411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575195.653415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283136:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575195.653422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff92fc4617d500 x1875449864282624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.653432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864282624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.653440:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.653443:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575195.653446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864282624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575195.653451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575195] req@ffff92fc4617d880 x1875449864282752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575195.653462:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864282752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575195.653469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575195.653472:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575195.653475:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864282752:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788575196.100872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575196.164811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575197.124873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575197.188836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575197.700797:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575197.700804:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575197.700806:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575197.700814:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6948 00000100:00080000:2.0:1788575197.700822:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575197.700829:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575197.700859:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864283264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575197.700883:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575197.700885:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575197.700887:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575197.700889:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6948 00000100:00080000:2.0:1788575197.700892:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575197.700895:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575197.700902:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575197.700911:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864283392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575197.700921:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864283264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575197.700925:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575197.700928:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575197.700929:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575197.700932:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6948 00000100:00080000:2.0:1788575197.700934:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575197.700936:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575197.700938:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575197.700949:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864283520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575197.700963:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575197.700966:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575197.700968:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575197.700971:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6948 00000100:00080000:2.0:1788575197.700975:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575197.700977:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575197.700981:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575197.700987:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575197.700999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864283648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575197.701008:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575197.701011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575197.701013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575197.701018:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6948 00000100:00100000:0.0:1788575197.701020:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff92fc4617c380 x1875449864283264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575197.701022:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575197.701030:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575197.701040:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575197.701047:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864283264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575197.701052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864283776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575197.701070:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.701076:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575197.701081:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864283264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575197.701106:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575197.701128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575197.701141:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff92fc4617d180 x1875449864283392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.701153:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864283392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.701162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.701165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575197.701167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575197.701172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575197.701190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575197.701200:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff92fc4617c000 x1875449864283520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.701210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864283520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.701217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.701220:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575197.701223:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575197.701228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575197.701248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575197.701258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff92fc4617fb80 x1875449864283648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.701266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864283648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.701273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.701275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575197.701278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575197.701283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575197.701300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575197.701309:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff92fc4617e680 x1875449864283776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.701319:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864283776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.701325:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.701328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575197.701331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864283776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575198.148836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575198.212828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575199.172851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575199.236825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575200.196875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575200.260813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575201.220899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575201.284861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575202.244902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575202.308839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575202.820842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575202.820859:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575202.820863:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575202.820872:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6953 00000100:00080000:3.0:1788575202.820882:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575202.820888:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575202.820921:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864283904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575202.820949:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575202.820952:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575202.820954:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575202.820958:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6953 00000100:00080000:3.0:1788575202.820962:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575202.820966:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575202.820976:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575202.820989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864284032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575202.820992:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864283904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575202.821007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575202.821010:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575202.821011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575202.821016:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6953 00000100:00080000:3.0:1788575202.821019:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575202.821023:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575202.821027:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575202.821036:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864284160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575202.821042:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575202.821045:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575202.821046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575202.821050:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6953 00000100:00080000:3.0:1788575202.821053:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575202.821058:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575202.821061:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575202.821068:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575202.821081:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864284288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575202.821094:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff92fc67339f80 x1875449864283904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575202.821096:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575202.821105:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575202.821110:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575202.821115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6953 00000100:00100000:0.0:1788575202.821118:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864283904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575202.821121:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575202.821128:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788575202.821131:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788575202.821134:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575202.821136:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575202.821141:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864283904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575202.821149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864284416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575202.821162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575202.821178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575202.821186:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff92fc6733ad80 x1875449864284032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.821193:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864284032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.821198:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.821200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575202.821202:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575202.821205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575202.821214:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575202.821220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff92fc67339180 x1875449864284160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.821227:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864284160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.821231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.821233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575202.821234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575202.821237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575202.821246:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575202.821251:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff92fc67339c00 x1875449864284288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.821258:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864284288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.821262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.821263:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575202.821265:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575202.821267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575202.821276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575202.821281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff92fc67339880 x1875449864284416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575202.821287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864284416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.821315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575202.821317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575202.821319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575203.268858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575203.332818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575204.292863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575204.356789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575205.316904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575205.380898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575206.340856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575206.404840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575207.364834:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575207.428806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575207.940845:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.940853:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575207.940857:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.940864:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6958 00000100:00080000:2.0:1788575207.940872:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.940878:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575207.940906:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864284544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.940927:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.940929:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575207.940931:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.940933:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6958 00000100:00080000:2.0:1788575207.940935:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.940937:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575207.940944:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575207.940952:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864284672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.940958:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.940962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575207.940963:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.940965:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6958 00000100:00080000:2.0:1788575207.940968:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575207.940972:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864284544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575207.940973:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575207.940980:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575207.940988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864284800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.940994:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.940996:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575207.940997:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.941000:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6958 00000100:00080000:2.0:1788575207.941002:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.941004:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575207.941006:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575207.941013:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864284928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575207.941018:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575207.941020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575207.941021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575207.941023:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6958 00000800:00000100:0.0:1788575207.941026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575207.941026:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575207.941030:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575207.941034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575207.941042:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864285056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575207.941062:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff92fc4617f800 x1875449864284544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.941074:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864284544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.941101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.941104:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575207.941107:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864284544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575207.941112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575207.941127:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575207.941135:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff92fc4617f100 x1875449864284672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.941142:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864284672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.941148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.941149:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575207.941151:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575207.941168:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575207.941178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575207.941184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff92fc4617dc00 x1875449864284800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.941190:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864284800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.941195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.941197:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575207.941199:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575207.941202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575207.941210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575207.941215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff92fc4617c700 x1875449864284928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.941221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864284928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.941225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.941227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575207.941229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864284928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575207.941232:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575207.941241:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575207.941246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575207] req@ffff92fc4617ca80 x1875449864285056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575207.941252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864285056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575207.941257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575207.941258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575207.941260:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575208.388839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575208.452795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575209.412838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575209.476799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575210.436845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575210.500803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575211.460873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575211.524802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575212.484921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575212.548823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575213.060855:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575213.060861:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575213.060873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575213.060881:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6963 00000100:00080000:2.0:1788575213.060890:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575213.060897:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575213.060927:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864285184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575213.060952:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575213.060955:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575213.060956:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575213.060959:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6963 00000100:00080000:2.0:1788575213.060964:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575213.060968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575213.060976:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575213.060986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864285312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575213.060992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575213.060995:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575213.060997:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575213.060999:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6963 00000100:00080000:2.0:1788575213.061001:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575213.061003:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575213.061006:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575213.061010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864285184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575213.061017:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864285440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575213.061023:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575213.061025:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575213.061028:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575213.061033:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6963 00000100:00080000:2.0:1788575213.061038:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575213.061040:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575213.061043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575213.061051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864285568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575213.061057:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575213.061060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575213.061061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575213.061064:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6963 00000100:00080000:2.0:1788575213.061067:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575213.061070:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575213.061073:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575213.061081:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575213.061087:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864285696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575213.061131:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575213/real 1788575213] req@ffff92fc4617ed80 x1875449864285184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575213.061150:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864285184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575213.061158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575213.061161:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575213.061164:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864285184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575213.061170:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575213.061257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575213.061267:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575213/real 1788575213] req@ffff92fc4617f480 x1875449864285312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575213.061384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864285312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575213.061394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575213.061399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575213.061404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575213.061469:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575213.061496:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575213.061521:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575213/real 1788575213] req@ffff92fc4617ca80 x1875449864285440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575213.061535:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864285440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575213.061542:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575213.061545:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575213.061548:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575213.061553:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575213.061573:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575213.061585:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575213/real 1788575213] req@ffff92fc4617c700 x1875449864285568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575213.061597:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864285568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575213.061610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575213.061613:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575213.061616:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575213.061622:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575213.061640:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575213.061651:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575213/real 1788575213] req@ffff92fc4617dc00 x1875449864285696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575213.061661:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864285696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575213.061668:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575213.061671:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575213.061674:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575213.508921:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575213.572900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575214.532878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575214.596913:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575215.556830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575215.620841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575216.580864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575216.644801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575217.604896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575217.668797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575218.180870:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575218.180881:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575218.180886:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575218.180896:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6968 00000100:00080000:3.0:1788575218.180905:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575218.180911:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575218.180942:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864285824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575218.180967:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575218.180971:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575218.180972:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575218.180974:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6968 00000100:00080000:3.0:1788575218.180977:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575218.180979:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575218.180987:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575218.180995:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864285952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575218.181002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575218.181004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575218.181006:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575218.181009:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6968 00000100:00080000:3.0:1788575218.181011:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575218.181014:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575218.181017:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575218.181023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864286080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575218.181031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575218.181032:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864285824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575218.181033:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575218.181036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575218.181042:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6968 00000100:00080000:3.0:1788575218.181045:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575218.181047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575218.181051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575218.181060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864286208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575218.181066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575218.181068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575218.181069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575218.181072:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6968 00000100:00080000:3.0:1788575218.181075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575218.181077:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575218.181079:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575218.181086:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864286336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575218.181094:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575218.181132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff92fc6733b800 x1875449864285824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575218.181149:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864285824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575218.181156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575218.181159:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575218.181162:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864285824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575218.181167:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575218.181181:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575218.181189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff92fc67338a80 x1875449864285952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575218.181196:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864285952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575218.181202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575218.181203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575218.181206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864285952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575218.181223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575218.181234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575218.181241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff92fc67338380 x1875449864286080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575218.181248:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864286080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575218.181258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575218.181260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575218.181262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575218.181265:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575218.181275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575218.181281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff92fc6733b480 x1875449864286208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575218.181288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864286208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575218.181293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575218.181295:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575218.181297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575218.181300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575218.181310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575218.181315:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575218/real 1788575218] req@ffff92fc6733aa00 x1875449864286336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575218.181322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864286336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575218.181327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575218.181328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575218.181331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575218.628880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575218.692817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575219.652873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575219.716842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575220.676800:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575220.740841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575221.700842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575221.764805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575222.724875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575222.788814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575223.300846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.300860:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575223.300865:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.300875:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6973 00000100:00080000:3.0:1788575223.300886:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.300894:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575223.300932:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864286464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.300966:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.300971:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575223.300973:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.300978:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6973 00000100:00080000:3.0:1788575223.300983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.300988:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575223.300999:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575223.301018:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864286464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575223.301020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864286592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.301036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.301039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575223.301041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.301044:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6973 00000100:00080000:3.0:1788575223.301048:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.301051:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575223.301056:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575223.301067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864286720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.301073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575223.301079:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575223.301079:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575223.301081:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575223.301086:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6973 00000100:00080000:3.0:1788575223.301089:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.301092:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575223.301096:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575223.301104:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864286848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575223.301115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575223.301125:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575223.301126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575223.301133:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff92fc6733bb80 x1875449864286464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575223.301133:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6973 00000100:00080000:3.0:1788575223.301139:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575223.301144:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575223.301151:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575223.301163:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864286976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575223.301172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864286464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575223.301195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575223.301200:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575223.301205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864286464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575223.301237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575223.301264:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575223.301280:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff92fc67338000 x1875449864286592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575223.301294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864286592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575223.301303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575223.301307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575223.301312:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575223.301319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575223.301341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575223.301353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff92fc67338700 x1875449864286720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575223.301366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864286720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575223.301375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575223.301378:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575223.301381:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575223.301387:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575223.301402:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575223.301409:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff92fc67339500 x1875449864286848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575223.301417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864286848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575223.301422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575223.301424:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575223.301426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575223.301430:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575223.301440:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575223.301465:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575223/real 1788575223] req@ffff92fc6733a300 x1875449864286976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575223.301478:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864286976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575223.301486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575223.301489:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575223.301493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864286976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575223.748883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575223.812803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575224.772870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575224.836839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575225.796872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575225.860823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575226.820850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575226.884851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575227.844905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575227.908792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575228.420842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.420852:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575228.420856:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.420867:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6979 00000100:00080000:2.0:1788575228.420878:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.420886:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575228.420923:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864287104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575228.420946:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.420950:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575228.420952:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.420956:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6979 00000100:00080000:2.0:1788575228.420961:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.420965:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575228.420981:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575228.420995:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864287104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575228.420995:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864287232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575228.421005:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.421008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575228.421010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.421013:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6979 00000100:00080000:2.0:1788575228.421016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.421019:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575228.421025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575228.421033:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864287360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575228.421041:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.421044:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575228.421046:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.421049:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6979 00000800:00000100:0.0:1788575228.421051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575228.421052:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575228.421055:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575228.421060:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575228.421068:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864287488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575228.421075:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575228.421078:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575228.421080:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575228.421083:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6979 00000100:00080000:2.0:1788575228.421086:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575228.421090:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff92fc4617f100 x1875449864287104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575228.421090:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575228.421094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575228.421102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864287616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575228.421108:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864287104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575228.421118:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575228.421121:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575228.421125:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864287104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575228.421147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575228.421172:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575228.421186:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff92fc4617f800 x1875449864287232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575228.421198:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864287232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575228.421205:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575228.421208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575228.421212:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575228.421217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575228.421268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575228.421281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff92fc4617e680 x1875449864287360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575228.421293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864287360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575228.421301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575228.421305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575228.421308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575228.421314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575228.421335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575228.421347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff92fc4617fb80 x1875449864287488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575228.421360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864287488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575228.421369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575228.421372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575228.421375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575228.421392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575228.421413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575228.421424:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575228/real 1788575228] req@ffff92fc4617c000 x1875449864287616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575228.421436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864287616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575228.421445:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575228.421449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575228.421452:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575228.868864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575228.932808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575229.892869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575229.956803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575230.916829:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575230.980816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575231.940872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575232.004823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575232.964821:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575233.028820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575233.540858:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575233.540867:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575233.540870:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575233.540878:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6984 00000100:00080000:0.0:1788575233.540887:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575233.540893:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575233.540926:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864287744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575233.540956:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575233.540960:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575233.540962:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575233.540967:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6984 00000100:00080000:0.0:1788575233.540971:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575233.540975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575233.540986:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575233.540998:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864287872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575233.541008:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864287744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575233.541009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575233.541013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575233.541014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575233.541017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6984 00000100:00080000:0.0:1788575233.541044:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575233.541048:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575233.541052:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575233.541062:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864288000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575233.541070:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575233.541072:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575233.541074:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575233.541077:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6984 00000100:00080000:0.0:1788575233.541080:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575233.541083:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575233.541092:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575233.541100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575233.541102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864288128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575233.541110:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575233.541112:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575233.541113:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575233.541116:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6984 00000100:00080000:0.0:1788575233.541118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575233.541121:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575233.541124:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575233.541131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864288256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575233.541156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff92fc67048000 x1875449864287744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575233.541177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864287744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575233.541187:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575233.541191:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575233.541195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864287744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575233.541217:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575233.541235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575233.541245:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff92fc6704aa00 x1875449864287872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575233.541253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864287872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575233.541259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575233.541261:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575233.541264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864287872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575233.541267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575233.541278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575233.541285:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff92fc6704a680 x1875449864288000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575233.541292:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864288000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575233.541297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575233.541299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575233.541301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575233.541304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575233.541315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575233.541321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff92fc6704ad80 x1875449864288128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575233.541328:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864288128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575233.541337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575233.541339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575233.541341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575233.541344:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575233.541354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575233.541359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575233/real 1788575233] req@ffff92fc67049880 x1875449864288256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575233.541366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864288256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575233.541371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575233.541373:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575233.541375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575233.988889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575234.052848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575235.012885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575235.076839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575236.036884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575236.100853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575237.060797:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575237.124816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575238.084917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575238.148876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575238.660799:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.660807:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575238.660825:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.660833:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6989 00000100:00080000:3.0:1788575238.660842:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575238.660848:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575238.660879:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864288384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.660900:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.660904:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575238.660905:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.660907:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6989 00000100:00080000:3.0:1788575238.660910:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575238.660912:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575238.660921:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575238.660930:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864288512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575238.660937:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864288384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575238.660940:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.660946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575238.660949:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.660952:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6989 00000100:00080000:3.0:1788575238.660955:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575238.660957:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575238.660960:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575238.660969:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864288640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.660975:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.660977:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575238.660979:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.660981:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6989 00000100:00080000:3.0:1788575238.660985:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575238.660987:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575238.660991:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575238.660998:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864288768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.661004:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575238.661007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575238.661008:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.661013:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6989 00000800:00000100:0.0:1788575238.661014:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575238.661016:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575238.661020:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575238.661024:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575238.661031:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864288896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575238.661058:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff92fc6733b100 x1875449864288384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575238.661078:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864288384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575238.661086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575238.661089:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575238.661091:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864288384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575238.661110:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575238.661128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575238.661137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff92fc6733a680 x1875449864288512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575238.661145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864288512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575238.661151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575238.661153:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575238.661155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575238.661159:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575238.661170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575238.661176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff92fc67338e00 x1875449864288640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575238.661183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864288640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575238.661193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575238.661195:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575238.661197:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575238.661201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575238.661212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575238.661218:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff92fc6733a300 x1875449864288768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575238.661225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864288768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575238.661230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575238.661231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575238.661233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575238.661237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575238.661247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575238.661253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff92fc67339500 x1875449864288896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575238.661260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864288896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575238.661264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575238.661266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575238.661269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864288896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575239.108834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575239.172865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575240.132929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575240.196894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575241.156894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575241.220892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575242.180900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575242.244824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575243.204929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575243.268886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575243.780897:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.780915:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575243.780921:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.780935:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6994 00000100:00080000:1.0:1788575243.780949:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.780959:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575243.781005:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864289024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.781030:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.781035:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575243.781037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.781041:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6994 00000100:00080000:1.0:1788575243.781045:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.781049:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575243.781062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575243.781077:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864289152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.781090:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575243.781097:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864289024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575243.781108:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575243.781110:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.781116:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6994 00000100:00080000:1.0:1788575243.781120:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.781124:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575243.781132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575243.781148:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864289280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.781159:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.781164:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575243.781166:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.781170:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6994 00000100:00080000:1.0:1788575243.781176:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575243.781180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575243.781185:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575243.781194:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575243.781216:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864289408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.781224:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.781227:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575243.781228:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.781231:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6994 00000100:00080000:1.0:1788575243.781235:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.781239:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575243.781243:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575243.781244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff92fc6728ad80 x1875449864289024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575243.781262:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864289536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575243.781271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864289024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.781287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575243.781293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575243.781298:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864289024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575243.781330:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575243.781356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575243.781370:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff92fc67289880 x1875449864289152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575243.781383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864289152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.781393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575243.781397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575243.781401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575243.781408:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575243.781426:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575243.781437:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff92fc6728b100 x1875449864289280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575243.781449:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864289280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.781459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575243.781462:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575243.781466:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575243.781472:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575243.781493:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575243.781505:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff92fc67288000 x1875449864289408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575243.781518:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864289408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.781526:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575243.781530:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575243.781533:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575243.781540:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575243.781560:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575243.781572:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff92fc67288e00 x1875449864289536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575243.781584:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864289536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575243.781593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575243.781596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575243.781637:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575244.228917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575244.292913:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575245.252927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575245.316830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575246.276841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575246.340863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575247.300839:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575247.364793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575248.324836:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575248.388828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575248.900939:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.900948:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575248.900953:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.900965:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 6999 00000100:00080000:1.0:1788575248.900977:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.900986:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575248.901025:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864289664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.901063:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.901067:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575248.901068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.901072:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6999 00000100:00080000:1.0:1788575248.901076:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.901079:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575248.901090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575248.901100:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864289792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.901108:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.901111:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575248.901113:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.901116:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6999 00000100:00080000:1.0:1788575248.901119:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.901122:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575248.901126:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575248.901133:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864289920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.901140:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.901143:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575248.901145:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.901148:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6999 00000100:00080000:1.0:1788575248.901151:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.901157:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575248.901160:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864289664:192.168.202.132@tcp:250:kworker.0 00000008:00100000:1.0:1788575248.901161:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575248.901172:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864290048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.901179:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.901182:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575248.901183:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.901186:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 6999 00000100:00080000:1.0:1788575248.901188:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.901190:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575248.901193:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575248.901202:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864290176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575248.901310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575248.901559:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff92fc6728aa00 x1875449864289664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.901581:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864289664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.901593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.901598:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575248.901602:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864289664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575248.901610:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575248.901637:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575248.901652:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff92fc6728b480 x1875449864289792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.901666:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864289792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.901677:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.901680:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575248.901683:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575248.901691:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575248.901711:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575248.901754:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff92fc6728a300 x1875449864289920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.901774:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864289920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.901783:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.901787:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575248.901791:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864289920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575248.901823:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575248.901846:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575248.901858:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff92fc67288e00 x1875449864290048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.901879:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864290048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.901887:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.901890:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575248.901894:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575248.901900:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575248.901921:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575248.901933:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff92fc67288000 x1875449864290176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.901946:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864290176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.901954:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.901958:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575248.901961:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575249.348892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575249.412859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575250.372846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575250.436858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575251.396888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575251.460919:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575252.420852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575252.484838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575253.444831:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575253.508850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575254.020825:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575254.020833:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575254.020837:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575254.020849:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7004 00000100:00080000:3.0:1788575254.020858:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575254.020865:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575254.020902:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864290304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575254.020934:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575254.020938:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575254.020940:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575254.020945:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7004 00000100:00080000:3.0:1788575254.020949:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575254.020954:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575254.020966:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575254.020981:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864290304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575254.020981:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864290432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575254.020997:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575254.021000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575254.021001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575254.021005:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7004 00000100:00080000:3.0:1788575254.021007:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575254.021010:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575254.021014:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575254.021023:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864290560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575254.021029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575254.021032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575254.021033:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575254.021035:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7004 00000100:00080000:3.0:1788575254.021039:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575254.021042:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575254.021046:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575254.021051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575254.021060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864290688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575254.021066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575254.021069:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575254.021070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575254.021073:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7004 00000100:00080000:3.0:1788575254.021076:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575254.021078:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575254.021082:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575254.021083:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575254/real 1788575254] req@ffff92fc67338700 x1875449864290304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575254.021091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864290816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575254.021103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864290304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575254.021112:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575254.021116:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575254.021120:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864290304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575254.021141:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575254.021164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575254.021208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575254/real 1788575254] req@ffff92fc67338000 x1875449864290432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575254.021224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864290432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575254.021233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575254.021235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575254.021237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575254.021242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575254.021259:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575254.021267:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575254/real 1788575254] req@ffff92fc6733bb80 x1875449864290560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575254.021275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864290560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575254.021280:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575254.021282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575254.021285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575254.021290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575254.021309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575254.021321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575254/real 1788575254] req@ffff92fc6733aa00 x1875449864290688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575254.021334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864290688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575254.021343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575254.021346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575254.021350:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575254.021356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575254.021377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575254.021388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575254/real 1788575254] req@ffff92fc6733b480 x1875449864290816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575254.021396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864290816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575254.021402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575254.021404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575254.021406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864290816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575254.468843:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575254.532845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575255.492849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575255.556839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575256.516817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575256.580823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575257.540910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575257.604817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575258.564871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575258.628868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575259.140928:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575259.140943:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575259.140949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575259.140962:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7009 00000100:00080000:0.0:1788575259.141008:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575259.141018:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575259.141067:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864290944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575259.141108:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575259.141114:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575259.141116:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575259.141122:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7009 00000100:00080000:0.0:1788575259.141126:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575259.141131:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575259.141142:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864290944:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788575259.141145:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575259.141163:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864291072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575259.141175:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575259.141179:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575259.141182:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575259.141186:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7009 00000100:00080000:0.0:1788575259.141191:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575259.141195:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575259.141203:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575259.141217:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864291200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575259.141234:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575259.141238:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575259.141239:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575259.141245:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7009 00000100:00080000:0.0:1788575259.141249:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575259.141254:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575259.141266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291072:192.168.202.132@tcp:38:kworker.0 00000008:00100000:0.0:1788575259.141268:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575259.141283:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864291328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575259.141292:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575259.141297:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575259.141298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291200:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788575259.141300:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575259.141306:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7009 00000100:00080000:0.0:1788575259.141310:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575259.141313:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575259.141317:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575259.141325:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575259.141329:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864291456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575259.141363:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575259.588916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575259.652800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575260.612939:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575260.676868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575261.636859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575261.700833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788575262.213032:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788575262.213082:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575262.213100:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575262.213154:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575262.213165:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575262.213172:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575262.213181:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788575262.213282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575259/real 1788575262] req@ffff92fc67049c00 x1875449864291456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575262.213318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864291456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575262.213329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575262.213334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575262.213338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291456:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575262.213346:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575259/real 1788575262] req@ffff92fc67049180 x1875449864291328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575262.213358:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864291328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575262.213365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575262.213368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575262.213371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575262.213378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575259/real 1788575262] req@ffff92fc6704bb80 x1875449864291072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575262.213390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864291072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575262.213398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575262.213402:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575262.213406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575262.213425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575259/real 1788575262] req@ffff92fc67049500 x1875449864291200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575262.213441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864291200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575262.213449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575262.213453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575262.213456:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575262.213463:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575259/real 1788575262] req@ffff92fc6704b100 x1875449864290944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575262.213475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864290944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575262.213489:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575262.213492:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575262.213495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864290944:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575262.660888:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575262.724831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575263.684876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575263.748855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575264.260872:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.260887:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575264.260892:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.260907:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7014 00000100:00080000:3.0:1788575264.260920:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.260931:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575264.260979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864291584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575264.261016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.261023:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575264.261026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.261031:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7014 00000100:00080000:3.0:1788575264.261036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.261041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575264.261054:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575264.261061:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864291584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575264.261073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864291712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575264.261087:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.261091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575264.261093:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.261098:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7014 00000100:00080000:3.0:1788575264.261102:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.261107:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575264.261112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575264.261126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864291840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575264.261137:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575264.261141:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575264.261143:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.261148:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7014 00000800:00000100:0.0:1788575264.261154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575264.261154:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575264.261163:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575264.261171:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575264.261196:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864291968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575264.261213:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575264.261215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff92fc67338380 x1875449864291584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575264.261219:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575264.261224:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575264.261240:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7014 00000100:00080000:3.0:1788575264.261246:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575264.261251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864291584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575264.261256:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575264.261267:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575264.261274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575264.261282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575264.261286:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864292096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575264.261292:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864291584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575264.261334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575264.261365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575264.261381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff92fc67338a80 x1875449864291712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575264.261395:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864291712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575264.261406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575264.261411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575264.261415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575264.261422:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575264.261443:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575264.261455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff92fc6733b800 x1875449864291840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575264.261477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864291840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575264.261486:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575264.261490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575264.261493:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575264.261500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575264.261523:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575264.261536:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff92fc67339880 x1875449864291968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575264.261550:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864291968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575264.261560:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575264.261565:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575264.261569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864291968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575264.261593:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575264.261619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575264.261632:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575264/real 1788575264] req@ffff92fc67339c00 x1875449864292096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575264.261645:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864292096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575264.261662:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575264.261667:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575264.261671:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575264.708936:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575264.772855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575265.732901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575265.796901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575266.756852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575266.820863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575267.780834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575267.844833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575268.804871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575268.868795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575269.380857:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575269.380868:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575269.380873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575269.380883:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7019 00000100:00080000:0.0:1788575269.380894:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575269.380902:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575269.380952:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864292224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575269.380981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575269.380986:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575269.380988:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575269.380992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7019 00000100:00080000:0.0:1788575269.380997:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575269.381002:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575269.381013:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575269.381023:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864292224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575269.381031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864292352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575269.381044:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575269.381048:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575269.381049:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575269.381053:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7019 00000100:00080000:0.0:1788575269.381057:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575269.381061:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575269.381066:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575269.381078:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864292480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575269.381089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575269.381093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575269.381095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575269.381100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575269.381103:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7019 00000100:00080000:0.0:1788575269.381111:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575269.381116:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575269.381122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575269.381136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864292608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575269.381146:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575269.381149:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff92fc67048e00 x1875449864292224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575269.381151:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575269.381156:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575269.381166:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7019 00000100:00080000:0.0:1788575269.381172:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575269.381178:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575269.381183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864292224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575269.381192:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575269.381201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575269.381210:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575269.381212:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864292736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575269.381216:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864292224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575269.381243:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575269.381265:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575269.381281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff92fc6704b100 x1875449864292352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575269.381289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864292352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575269.381295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575269.381298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575269.381300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575269.381304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575269.381316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575269.381332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff92fc67049500 x1875449864292480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575269.381345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864292480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575269.381354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575269.381357:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575269.381361:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575269.381367:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575269.381390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575269.381402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff92fc6704bb80 x1875449864292608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575269.381422:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864292608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575269.381431:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575269.381435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575269.381439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575269.381445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575269.381468:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575269.381479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575269/real 1788575269] req@ffff92fc67049180 x1875449864292736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575269.381487:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864292736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575269.381492:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575269.381494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575269.381496:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575269.828843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575269.892830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575270.852932:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575270.916871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575271.876979:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575271.940875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575272.900830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575272.964864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575273.924824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575273.988789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575274.500861:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575274.500872:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575274.500876:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575274.500884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7025 00000100:00080000:3.0:1788575274.500894:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575274.500901:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575274.500935:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864292864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575274.500967:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575274.500971:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575274.500973:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575274.500977:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7025 00000100:00080000:3.0:1788575274.500981:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575274.500986:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575274.500995:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575274.501007:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864292864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575274.501010:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864292992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575274.501023:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575274.501025:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575274.501026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575274.501029:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7025 00000100:00080000:3.0:1788575274.501032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575274.501034:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575274.501037:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575274.501044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864293120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575274.501050:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575274.501052:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575274.501053:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575274.501056:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7025 00000100:00080000:3.0:1788575274.501059:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575274.501061:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575274.501064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575274.501073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864293248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575274.501082:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575274.501086:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575274.501091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575274.501093:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575274.501097:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7025 00000100:00080000:3.0:1788575274.501101:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575274.501105:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575274.501110:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575274.501122:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864293376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575274.501141:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff92fc67339180 x1875449864292864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575274.501161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864292864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.501177:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575274.501181:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575274.501185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864292864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575274.501207:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575274.501228:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575274.501241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff92fc6733ad80 x1875449864292992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575274.501268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864292992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.501278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575274.501281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575274.501283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864292992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575274.501289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575274.501309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575274.501320:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff92fc67339f80 x1875449864293120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575274.501331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864293120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.501339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575274.501342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575274.501344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575274.501349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575274.501368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575274.501377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff92fc67339c00 x1875449864293248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575274.501388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864293248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.501400:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575274.501404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575274.501407:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575274.501412:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575274.501470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575274.501482:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575274/real 1788575274] req@ffff92fc67339880 x1875449864293376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575274.501492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864293376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575274.501499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575274.501502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575274.501505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575274.948832:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575275.012780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575275.972823:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575276.036814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575276.996849:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575277.060846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575278.020910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575278.084896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575279.044863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575279.108816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575279.620839:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.620846:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575279.620849:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.620856:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7030 00000100:00080000:2.0:1788575279.620864:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.620869:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575279.620896:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864293504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.620916:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.620919:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.620920:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.620922:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7030 00000100:00080000:2.0:1788575279.620924:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.620926:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575279.620933:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575279.620941:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864293632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.620950:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.620953:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575279.620955:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.620959:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7030 00000100:00080000:2.0:1788575279.620963:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575279.620966:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864293504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575279.620970:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575279.620976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575279.620989:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864293760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.620997:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.620999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.621000:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.621002:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7030 00000100:00080000:2.0:1788575279.621005:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.621008:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575279.621011:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575279.621016:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575279.621020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864293888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.621026:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.621028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575279.621030:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.621032:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7030 00000100:00080000:2.0:1788575279.621034:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.621036:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575279.621039:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575279.621046:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864294016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575279.621053:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff92fc4617d180 x1875449864293504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575279.621067:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864293504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575279.621075:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575279.621078:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575279.621080:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864293504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575279.621085:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575279.621097:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575279.621105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff92fc4617c380 x1875449864293632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575279.621111:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864293632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575279.621116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575279.621118:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575279.621120:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575279.621137:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575279.621147:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575279.621153:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff92fc4617d880 x1875449864293760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575279.621159:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864293760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575279.621164:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575279.621165:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575279.621167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575279.621170:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575279.621179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575279.621185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff92fc4617d500 x1875449864293888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575279.621191:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864293888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575279.621202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575279.621204:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575279.621206:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864293888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575279.621209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575279.621219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575279.621225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff92fc4617e300 x1875449864294016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575279.621231:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864294016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575279.621236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575279.621238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575279.621239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575280.068863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575280.132801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575281.092857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575281.156853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575282.116917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575282.180830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575283.140894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575283.204830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575284.164863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575284.228800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575284.740826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.740835:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575284.740838:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.740845:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7035 00000100:00080000:3.0:1788575284.740854:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.740860:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575284.740889:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864294144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575284.740911:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.740916:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575284.740917:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.740920:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7035 00000100:00080000:3.0:1788575284.740923:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.740926:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575284.740933:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575284.740940:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864294144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575284.740945:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864294272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575284.740954:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.740956:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575284.740957:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.740959:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7035 00000100:00080000:3.0:1788575284.740960:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.740962:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575284.740965:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575284.740971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864294400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575284.740977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.740979:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575284.740980:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.740982:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7035 00000100:00080000:3.0:1788575284.740993:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.740997:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575284.741001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575284.741009:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575284.741011:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864294528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575284.741018:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.741020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575284.741021:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.741024:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7035 00000100:00080000:3.0:1788575284.741027:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.741030:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575284.741032:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575284.741039:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864294656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575284.741053:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff92fc6733b800 x1875449864294144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575284.741067:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864294144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575284.741075:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575284.741078:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575284.741081:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864294144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575284.741112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575284.741129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575284.741137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff92fc67338a80 x1875449864294272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575284.741144:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864294272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575284.741149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575284.741151:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575284.741153:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575284.741157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575284.741169:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575284.741178:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff92fc67338380 x1875449864294400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575284.741189:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864294400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575284.741197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575284.741201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575284.741205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575284.741211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575284.741229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575284.741239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff92fc6733b480 x1875449864294528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575284.741250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864294528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575284.741259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575284.741262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575284.741265:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575284.741271:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575284.741286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575284.741294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff92fc6733aa00 x1875449864294656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575284.741306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864294656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575284.741312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575284.741315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575284.741317:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575285.188900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575285.252826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575286.212853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575286.276815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575287.236881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575287.300890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575288.260911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575288.324821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575289.284846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575289.348830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575289.860826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.860835:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575289.860838:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.860846:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7040 00000100:00080000:3.0:1788575289.860855:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.860861:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575289.860895:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864294784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.860942:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.860947:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575289.860950:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.860955:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7040 00000100:00080000:3.0:1788575289.860961:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.860966:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575289.860973:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864294784:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788575289.860976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575289.860993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864294912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.861002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.861004:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575289.861006:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.861009:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7040 00000100:00080000:3.0:1788575289.861011:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.861014:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575289.861017:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575289.861026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864295040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.861032:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.861035:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575289.861036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.861038:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7040 00000100:00080000:3.0:1788575289.861040:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.861044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788575289.861047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788575289.861049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575289.861059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864295168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.861070:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.861072:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575289.861074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.861075:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7040 00000100:00080000:3.0:1788575289.861077:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.861080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575289.861083:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575289.861091:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864295296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575289.861100:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff92fc6733bb80 x1875449864294784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575289.861123:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864294784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575289.861135:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575289.861140:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575289.861143:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864294784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575289.861164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575289.861183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575289.861193:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff92fc67338000 x1875449864294912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575289.861201:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864294912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575289.861212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575289.861214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575289.861217:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864294912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575289.861221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575289.861232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575289.861238:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff92fc67338700 x1875449864295040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575289.861246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864295040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575289.861251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575289.861253:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575289.861255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575289.861258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575289.861269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575289.861275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff92fc67339500 x1875449864295168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575289.861282:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864295168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575289.861287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575289.861289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575289.861291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575289.861295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575289.861305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575289.861311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff92fc6733a300 x1875449864295296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575289.861317:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864295296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575289.861323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575289.861324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575289.861335:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575290.308878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575290.372826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575291.332895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575291.396832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575292.356917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575292.420871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575293.380831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575293.444867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575294.404930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575294.468877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575294.980830:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.980842:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575294.980847:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.980857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7045 00000100:00080000:3.0:1788575294.980868:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.980876:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575294.980913:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864295424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575294.980945:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.980950:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575294.980952:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.980956:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7045 00000100:00080000:3.0:1788575294.980959:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.980964:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575294.980975:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575294.980990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864295552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575294.980994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864295424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575294.981023:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.981028:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575294.981030:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.981035:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7045 00000100:00080000:3.0:1788575294.981042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.981047:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788575294.981053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788575294.981053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575294.981070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864295680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575294.981081:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.981085:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575294.981087:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.981094:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7045 00000100:00100000:2.0:1788575294.981095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff92fc67338e00 x1875449864295424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575294.981098:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.981106:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575294.981116:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575294.981127:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864295424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575294.981132:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864295808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575294.981151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575294.981153:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.981159:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575294.981160:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788575294.981165:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575294.981169:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864295424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575294.981173:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7045 00000100:00080000:3.0:1788575294.981181:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.981185:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575294.981191:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575294.981202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575294.981206:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864295936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575294.981234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575294.981253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff92fc6733a680 x1875449864295552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575294.981269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864295552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575294.981278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575294.981282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575294.981285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575294.981290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575294.981306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575294.981322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff92fc6733b100 x1875449864295680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575294.981331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864295680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575294.981337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575294.981339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575294.981341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575294.981345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575294.981356:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575294.981363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff92fc6733a300 x1875449864295808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575294.981370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864295808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575294.981375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575294.981377:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575294.981379:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575294.981393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575294.981420:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575294.981430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff92fc67339500 x1875449864295936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575294.981442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864295936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575294.981450:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575294.981453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575294.981457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864295936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575295.428825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575295.492844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575296.452887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575296.516834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575297.476851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575297.540829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575298.500855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575298.564830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575299.524887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575299.588819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575300.100890:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575300.100904:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575300.100909:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575300.100922:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7050 00000100:00080000:3.0:1788575300.100935:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575300.100944:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575300.100990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864296064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575300.101020:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575300.101024:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575300.101025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575300.101029:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7050 00000100:00080000:3.0:1788575300.101033:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575300.101036:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575300.101048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575300.101060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864296192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575300.101075:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575300.101078:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575300.101079:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575300.101083:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7050 00000100:00080000:3.0:1788575300.101086:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575300.101090:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575300.101096:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575300.101111:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864296064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575300.101111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864296320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575300.101135:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575300.101140:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575300.101143:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575300.101147:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7050 00000100:00080000:3.0:1788575300.101151:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575300.101155:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575300.101161:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575300.101174:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864296448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575300.101183:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575300.101190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575300.101198:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575300.101202:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575300.101209:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7050 00000100:00080000:3.0:1788575300.101214:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575300.101219:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575300.101225:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575300.101240:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864296576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575300.101250:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575300/real 1788575300] req@ffff92fc67338700 x1875449864296064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575300.101270:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864296064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575300.101279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575300.101285:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575300.101289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864296064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575300.101298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575300.101316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575300.101324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575300/real 1788575300] req@ffff92fc67338000 x1875449864296192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575300.101332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864296192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575300.101339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575300.101341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575300.101343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575300.101364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575300.101384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575300.101395:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575300/real 1788575300] req@ffff92fc6733bb80 x1875449864296320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575300.101409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864296320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575300.101418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575300.101421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575300.101423:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575300.101428:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575300.101444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575300.101452:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575300/real 1788575300] req@ffff92fc6733aa00 x1875449864296448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575300.101460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864296448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575300.101466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575300.101468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575300.101470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575300.101473:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575300.101485:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575300.101491:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575300/real 1788575300] req@ffff92fc6733b480 x1875449864296576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575300.101498:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864296576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575300.101503:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575300.101505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575300.101507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575300.548926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575300.612869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575301.572852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575301.636820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575302.596846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575302.660915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575303.620887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575303.684776:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575304.644892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575304.708836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575305.220899:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575305.220914:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575305.220920:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575305.220934:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7055 00000100:00080000:2.0:1788575305.220948:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575305.220957:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575305.221010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864296704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575305.221050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575305.221055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575305.221057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575305.221062:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7055 00000100:00080000:2.0:1788575305.221066:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575305.221069:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575305.221081:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575305.221097:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864296832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575305.221105:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864296704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575305.221122:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575305.221128:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575305.221130:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575305.221135:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7055 00000100:00080000:2.0:1788575305.221139:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575305.221143:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575305.221148:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575305.221161:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864296960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575305.221169:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575305.221171:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575305.221173:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575305.221179:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575305.221179:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7055 00000100:00080000:2.0:1788575305.221187:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575305.221191:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575305.221198:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575305.221210:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864297088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575305.221218:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575305.221220:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575305.221222:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575305.221225:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7055 00000100:00080000:2.0:1788575305.221228:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575305.221233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff92fc4617ce00 x1875449864296704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575305.221234:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575305.221241:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575305.221254:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864297216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575305.221268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864296704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.221282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.221286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575305.221290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864296704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575305.221317:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575305.221344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575305.221359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff92fc4617ea00 x1875449864296832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.221373:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864296832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.221383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.221387:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575305.221390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575305.221397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575305.221433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575305.221476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff92fc4617e300 x1875449864296960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.221491:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864296960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.221501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.221505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575305.221508:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864296960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575305.221514:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575305.221541:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575305.221555:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff92fc4617d500 x1875449864297088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.221569:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864297088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.221578:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.221581:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575305.221585:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575305.221592:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575305.221615:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575305.221628:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575305/real 1788575305] req@ffff92fc4617d880 x1875449864297216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575305.221640:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864297216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575305.221648:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575305.221650:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575305.221653:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575305.668820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575305.732789:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575306.692866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575306.756895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575307.716872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575307.780813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575308.740889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575308.804835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575309.764899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575309.828885:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575310.340830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.340841:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575310.340845:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.340855:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7060 00000100:00080000:3.0:1788575310.340866:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.340874:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575310.340926:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864297344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575310.340948:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.340952:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575310.340953:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.340956:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7060 00000100:00080000:3.0:1788575310.340959:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.340961:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575310.340972:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575310.340983:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864297472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575310.340991:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.340997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575310.340999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.341005:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7060 00000100:00100000:0.0:1788575310.341006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864297344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575310.341010:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.341024:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575310.341031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575310.341046:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864297600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575310.341058:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.341061:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575310.341062:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.341066:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7060 00000100:00080000:3.0:1788575310.341070:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.341073:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575310.341077:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575310.341083:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575310.341088:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864297728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575310.341095:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575310.341098:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575310.341100:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575310.341103:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7060 00000100:00080000:3.0:1788575310.341105:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575310.341108:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575310.341112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575310.341120:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864297856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575310.341142:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff92fc67338380 x1875449864297344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.341160:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864297344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.341170:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.341174:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575310.341177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864297344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575310.341183:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575310.341207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575310.341218:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff92fc67338a80 x1875449864297472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.341226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864297472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.341233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.341235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575310.341237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575310.341260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575310.341273:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575310.341295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff92fc6733b800 x1875449864297600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.341309:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864297600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.341315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.341317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575310.341319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575310.341322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575310.341336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575310.341343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff92fc67339880 x1875449864297728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.341351:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864297728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.341362:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.341364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575310.341366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575310.341370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575310.341382:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575310.341389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575310/real 1788575310] req@ffff92fc67339c00 x1875449864297856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575310.341396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864297856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575310.341406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575310.341408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575310.341410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864297856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575310.788842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575310.852896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575311.812902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575311.876867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575312.836833:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575312.900891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575313.860899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575313.924848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575314.884818:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575314.948805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575315.460874:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575315.460885:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575315.460890:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575315.460902:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7066 00000100:00080000:3.0:1788575315.460912:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575315.460920:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575315.460952:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864297984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575315.460975:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575315.460978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575315.460980:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575315.460982:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7066 00000100:00080000:3.0:1788575315.460985:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575315.460987:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575315.460995:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575315.461003:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864298112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575315.461011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575315.461014:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575315.461016:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575315.461021:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7066 00000100:00080000:3.0:1788575315.461025:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575315.461029:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575315.461034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575315.461043:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864297984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575315.461049:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864298240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575315.461061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575315.461065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575315.461067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575315.461071:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7066 00000100:00080000:3.0:1788575315.461075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575315.461079:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575315.461084:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575315.461096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864298368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575315.461106:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575315.461109:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575315.461112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575315.461129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575315.461148:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7066 00000100:00080000:3.0:1788575315.461154:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575315.461158:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575315.461163:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575315.461177:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864298496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575315.461194:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff92fc67339f80 x1875449864297984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.461218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864297984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.461230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.461235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575315.461239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864297984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575315.461247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575315.461279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575315.461294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff92fc6733ad80 x1875449864298112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.461307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864298112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.461317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.461321:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575315.461324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575315.461351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575315.461374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575315.461387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff92fc67339180 x1875449864298240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.461399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864298240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.461407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.461411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575315.461414:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575315.461421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575315.461442:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575315.461455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff92fc67339c00 x1875449864298368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.461468:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864298368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.461477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.461481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575315.461484:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575315.461491:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575315.461512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575315.461524:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575315/real 1788575315] req@ffff92fc67339880 x1875449864298496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575315.461538:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864298496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575315.461546:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575315.461550:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575315.461554:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575315.908866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575315.972845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575316.932853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575316.996841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575317.956923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575318.020869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575318.980947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575319.044860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575320.004934:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575320.068824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575320.580828:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.580839:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575320.580844:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.580854:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7071 00000100:00080000:3.0:1788575320.580865:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.580873:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575320.580910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864298624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.580939:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.580944:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575320.580946:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.580951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7071 00000100:00080000:3.0:1788575320.580968:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.580972:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575320.580983:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575320.580997:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864298752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.581022:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575320.581025:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864298624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575320.581027:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575320.581032:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.581042:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7071 00000100:00080000:3.0:1788575320.581048:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.581054:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575320.581061:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575320.581076:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864298880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.581087:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.581091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575320.581093:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.581098:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7071 00000100:00080000:3.0:1788575320.581102:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575320.581103:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575320.581111:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575320.581118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575320.581131:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864299008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575320.581149:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff92fc6733b800 x1875449864298624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.581166:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864298624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575320.581204:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.581208:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575320.581210:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575320.581217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575320.581218:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7071 00000100:00080000:3.0:1788575320.581225:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575320.581226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:3.0:1788575320.581233:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575320.581234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864298624:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788575320.581241:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575320.581254:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864299136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575320.581259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575320.581286:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575320.581297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff92fc67338a80 x1875449864298752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.581305:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864298752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.581311:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.581313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575320.581316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575320.581320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575320.581330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575320.581336:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff92fc67338380 x1875449864298880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.581348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864298880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.581353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.581355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575320.581357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864298880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575320.581360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575320.581371:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575320.581377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff92fc6733b480 x1875449864299008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.581384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864299008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.581389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.581391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575320.581393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575320.581402:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575320.581412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575320.581418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff92fc6733aa00 x1875449864299136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575320.581425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864299136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575320.581435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575320.581437:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575320.581439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575321.028890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575321.092822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575322.052878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575322.116858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575323.076901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575323.140830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575324.100885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575324.164872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575325.124944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575325.188824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575325.700826:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575325.700838:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575325.700843:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575325.700854:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7076 00000100:00080000:3.0:1788575325.700864:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575325.700873:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575325.700911:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864299264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575325.700943:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575325.700946:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575325.700947:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575325.700953:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7076 00000100:00080000:3.0:1788575325.700957:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575325.700961:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575325.700972:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575325.700989:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864299264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575325.700990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864299392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575325.701008:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575325.701011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575325.701014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575325.701018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7076 00000100:00080000:3.0:1788575325.701023:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575325.701028:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575325.701034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575325.701047:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864299520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575325.701062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575325.701065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575325.701066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575325.701069:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7076 00000100:00080000:3.0:1788575325.701072:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575325.701075:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575325.701079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575325.701089:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864299648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575325.701097:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575325.701099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299392:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788575325.701101:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575325.701104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575325.701108:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7076 00000100:00080000:3.0:1788575325.701110:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575325.701113:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575325.701118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575325.701127:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864299776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575325.701131:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575325.701157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575325.701173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575326.148896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575326.212839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575327.172944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575327.236893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575328.196865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575328.260830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788575328.773027:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788575328.773070:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575328.773086:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575328.773128:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575328.773137:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575328.773146:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575328.773154:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:2.0:1788575328.773275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff92fc67339500 x1875449864299648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575328.773300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864299648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575328.773312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575328.773316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575328.773320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575328.773327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff92fc6733a300 x1875449864299776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575328.773339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864299776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575328.773346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575328.773350:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575328.773353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299776:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575328.773358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff92fc67338000 x1875449864299392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575328.773369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864299392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575328.773376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575328.773380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575328.773383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575328.773389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff92fc67338700 x1875449864299520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575328.773399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864299520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575328.773406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575328.773409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575328.773412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864299520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575328.773418:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575328] req@ffff92fc6733bb80 x1875449864299264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575328.773436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864299264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575328.773443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575328.773446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575328.773449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864299264:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575329.220875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575329.284801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575330.244857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575330.308812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575330.820811:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.820824:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575330.820829:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.820843:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7081 00000100:00080000:3.0:1788575330.820858:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.820867:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575330.820910:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864299904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575330.820943:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.820947:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575330.820949:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.820953:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7081 00000100:00080000:3.0:1788575330.820957:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.820961:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575330.820973:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575330.820985:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864300032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575330.820998:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.821002:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575330.821004:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.821008:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7081 00000100:00080000:3.0:1788575330.821013:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.821019:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575330.821025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575330.821036:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864299904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575330.821044:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864300160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575330.821060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.821065:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575330.821067:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.821071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7081 00000100:00080000:3.0:1788575330.821074:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.821078:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575330.821084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575330.821107:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864300288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575330.821114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575330.821131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575330.821141:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575330.821143:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575330.821149:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7081 00000100:00080000:3.0:1788575330.821155:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575330.821162:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575330.821169:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575330.821186:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864300416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575330.821194:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff92fc6733b100 x1875449864299904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575330.821230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864299904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575330.821245:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575330.821252:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575330.821257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864299904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575330.821267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575330.821298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575330.821312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff92fc6733a680 x1875449864300032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575330.821323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864300032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575330.821333:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575330.821337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575330.821341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575330.821373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575330.821397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575330.821410:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff92fc67338e00 x1875449864300160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575330.821423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864300160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575330.821432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575330.821435:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575330.821438:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575330.821443:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575330.821466:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575330.821478:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff92fc6733bb80 x1875449864300288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575330.821490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864300288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575330.821498:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575330.821501:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575330.821505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575330.821519:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575330.821542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575330.821553:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff92fc67338700 x1875449864300416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575330.821564:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864300416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575330.821572:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575330.821575:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575330.821578:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575331.268886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575331.332865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575332.292809:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575332.356831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575333.316852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575333.380841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575334.340890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575334.404847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575335.364890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575335.428846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575335.940840:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.940855:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575335.940860:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.940874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7086 00000100:00080000:3.0:1788575335.940887:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.940897:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575335.940950:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864300544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575335.940983:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.940986:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575335.940988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.940991:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7086 00000100:00080000:3.0:1788575335.940994:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.940997:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575335.941007:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575335.941019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864300672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575335.941028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575335.941031:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864300544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575335.941032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575335.941036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.941041:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7086 00000100:00080000:3.0:1788575335.941043:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.941045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575335.941048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575335.941065:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864300800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575335.941072:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.941074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575335.941076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.941079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7086 00000100:00080000:3.0:1788575335.941091:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.941094:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575335.941098:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575335.941107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575335.941108:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864300928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575335.941116:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575335.941119:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575335.941121:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575335.941123:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7086 00000100:00080000:3.0:1788575335.941126:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575335.941129:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575335.941132:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575335.941145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864301056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575335.941164:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff92fc67338000 x1875449864300544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575335.941191:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864300544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575335.941201:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575335.941205:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575335.941208:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864300544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575335.941232:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575335.941249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575335.941258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff92fc6733a300 x1875449864300672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575335.941271:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864300672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575335.941278:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575335.941280:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575335.941283:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575335.941286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575335.941297:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575335.941303:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff92fc67339500 x1875449864300800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575335.941316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864300800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575335.941321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575335.941324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575335.941326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575335.941329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575335.941341:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575335.941347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff92fc6733aa00 x1875449864300928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575335.941355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864300928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575335.941365:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575335.941367:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575335.941369:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864300928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575335.941373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575335.941383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575335.941388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff92fc6733b480 x1875449864301056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575335.941395:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864301056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575335.941401:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575335.941403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575335.941405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575336.388858:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575336.452812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575337.412900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575337.476866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575338.436882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575338.500806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575339.460893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575339.524823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575340.484900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575340.548845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575341.060891:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575341.060906:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575341.060912:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575341.060926:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7091 00000100:00080000:3.0:1788575341.060941:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575341.060952:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575341.061004:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864301184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575341.061042:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575341.061050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575341.061052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575341.061057:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7091 00000100:00080000:3.0:1788575341.061061:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575341.061066:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575341.061080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575341.061099:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864301312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575341.061112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864301184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575341.061115:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575341.061124:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575341.061126:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575341.061132:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7091 00000100:00080000:3.0:1788575341.061136:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575341.061142:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575341.061149:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575341.061165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864301440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575341.061177:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575341.061181:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575341.061183:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575341.061188:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7091 00000100:00080000:3.0:1788575341.061193:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575341.061198:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575341.061205:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575341.061212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575341.061221:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864301568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575341.061234:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575341.061238:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575341.061240:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575341.061245:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7091 00000100:00080000:3.0:1788575341.061249:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575341.061253:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575341.061259:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575341.061273:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864301696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575341.061278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575341/real 1788575341] req@ffff92fc67338380 x1875449864301184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575341.061310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864301184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575341.061324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575341.061329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575341.061334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864301184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575341.061370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575341.061397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575341.061412:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575341/real 1788575341] req@ffff92fc67338a80 x1875449864301312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575341.061427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864301312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575341.061437:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575341.061441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575341.061445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575341.061452:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575341.061475:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575341.061522:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575341/real 1788575341] req@ffff92fc6733b800 x1875449864301440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575341.061535:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864301440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575341.061545:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575341.061548:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575341.061552:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575341.061558:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575341.061581:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575341.061592:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575341/real 1788575341] req@ffff92fc67339880 x1875449864301568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575341.061605:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864301568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575341.061614:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575341.061617:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575341.061621:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575341.061628:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575341.061650:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575341.061662:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575341/real 1788575341] req@ffff92fc67339c00 x1875449864301696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575341.061675:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864301696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575341.061684:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575341.061688:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575341.061692:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575341.508929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575341.572837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575342.532846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575342.596793:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575343.556889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575343.620828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575344.580844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575344.644821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575345.604862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575345.668797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575346.180836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.180848:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575346.180852:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.180862:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7096 00000100:00080000:3.0:1788575346.180871:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.180878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575346.180910:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864301824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575346.180933:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.180936:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575346.180938:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.180940:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7096 00000100:00080000:3.0:1788575346.180943:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.180946:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575346.180955:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575346.180967:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864301952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575346.180977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.180980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575346.180982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.180987:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7096 00000100:00080000:3.0:1788575346.180991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.180997:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575346.181002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575346.181017:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864302080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575346.181018:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864301824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575346.181033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.181035:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575346.181037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575346.181040:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7096 00000100:00080000:3.0:1788575346.181043:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.181045:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575346.181049:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575346.181062:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864302208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575346.181078:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575346.181082:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575346.181085:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575346.181087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575346.181102:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7096 00000100:00080000:3.0:1788575346.181107:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575346.181111:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575346.181115:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575346.181126:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864302336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575346.181142:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff92fc67339180 x1875449864301824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575346.181158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864301824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575346.181171:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575346.181175:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575346.181178:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864301824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575346.181184:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575346.181199:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575346.181208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff92fc6733ad80 x1875449864301952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575346.181216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864301952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575346.181222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575346.181224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575346.181226:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864301952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575346.181246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575346.181256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575346.181263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff92fc67339f80 x1875449864302080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575346.181275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864302080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575346.181283:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575346.181286:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575346.181290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575346.181297:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575346.181317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575346.181327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff92fc67339c00 x1875449864302208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575346.181339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864302208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575346.181347:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575346.181350:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575346.181354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575346.181360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575346.181377:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575346.181387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575346/real 1788575346] req@ffff92fc67339880 x1875449864302336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575346.181398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864302336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575346.181424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575346.181428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575346.181431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575346.628867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575346.692827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575347.652821:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575347.716818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575348.676848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575348.740810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575349.700804:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575349.764798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575350.724871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575350.788799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575351.300803:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575351.300812:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575351.300816:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575351.300832:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7101 00000100:00080000:0.0:1788575351.300840:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575351.300847:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575351.300879:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864302464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575351.300900:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575351.300905:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575351.300906:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575351.300909:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7101 00000100:00080000:0.0:1788575351.300911:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575351.300913:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575351.300921:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575351.300929:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864302592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575351.300937:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575351.300939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575351.300941:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575351.300943:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7101 00000100:00080000:0.0:1788575351.300945:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575351.300948:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575351.300951:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575351.300953:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864302464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575351.300961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864302720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575351.300969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575351.300972:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575351.300973:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575351.300976:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7101 00000100:00080000:0.0:1788575351.300978:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575351.300980:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575351.300985:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575351.300992:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864302848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575351.300999:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575351.301001:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575351.301003:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575351.301004:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7101 00000100:00080000:0.0:1788575351.301007:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575351.301009:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:2.0:1788575351.301014:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788575351.301016:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575351.301029:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864302976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575351.301060:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff92fc67049c00 x1875449864302464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575351.301077:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864302464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575351.301086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575351.301090:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575351.301093:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864302464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575351.301098:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575351.301114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575351.301122:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff92fc67049880 x1875449864302592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575351.301131:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864302592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575351.301136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575351.301139:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575351.301141:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575351.301159:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:2.0:1788575351.301170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575351.301177:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff92fc6704ad80 x1875449864302720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575351.301184:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864302720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575351.301194:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575351.301196:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575351.301198:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575351.301202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575351.301212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575351.301218:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff92fc6704a680 x1875449864302848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575351.301225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864302848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575351.301230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575351.301232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575351.301234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:2.0:1788575351.301238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:2.0:1788575351.301247:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575351.301253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575351/real 1788575351] req@ffff92fc6704aa00 x1875449864302976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575351.301259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864302976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575351.301266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575351.301270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575351.301274:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864302976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575351.748833:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575351.812848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575352.772844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575352.836829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575353.796808:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575353.860846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575354.820873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575354.884825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575355.844855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575355.908805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575356.420836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575356.420844:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575356.420848:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575356.420857:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7107 00000100:00080000:3.0:1788575356.420866:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575356.420872:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575356.420905:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864303104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575356.420938:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575356.420943:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575356.420945:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575356.420950:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7107 00000100:00080000:3.0:1788575356.420954:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575356.420958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575356.420968:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575356.420982:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864303232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575356.420989:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864303104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575356.420995:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575356.420999:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575356.421000:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575356.421005:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7107 00000100:00080000:3.0:1788575356.421011:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575356.421016:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575356.421022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575356.421036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864303360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575356.421048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575356.421053:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575356.421056:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575356.421060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575356.421065:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7107 00000100:00080000:3.0:1788575356.421069:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575356.421072:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575356.421077:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575356.421088:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864303488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575356.421092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff92fc6733b800 x1875449864303104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575356.421102:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575356.421116:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864303104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575356.421124:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575356.421127:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575356.421129:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864303104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575356.421148:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303232:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788575356.421158:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575356.421160:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575356.421166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575356.421167:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7107 00000100:00080000:3.0:1788575356.421172:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575356.421174:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575356.421179:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575356.421190:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864303616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575356.421200:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff92fc67338a80 x1875449864303232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575356.421217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864303232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575356.421227:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575356.421231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575356.421233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575356.421238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575356.421274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575356.421284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff92fc67338380 x1875449864303360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575356.421291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864303360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575356.421302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575356.421304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575356.421307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575356.421312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575356.421326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575356.421334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff92fc6733b480 x1875449864303488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575356.421345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864303488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575356.421351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575356.421353:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575356.421356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575356.421371:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575356.421385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575356.421392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575356/real 1788575356] req@ffff92fc6733aa00 x1875449864303616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575356.421402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864303616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575356.421411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575356.421415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575356.421419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575356.868867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575356.932827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575357.892913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575357.956816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575358.916865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575358.980824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575359.940894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575360.004799:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575360.964856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575361.028845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575361.540856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575361.540865:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575361.540868:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575361.540877:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7112 00000100:00080000:1.0:1788575361.540886:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575361.540892:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575361.540928:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864303744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575361.540951:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575361.540954:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575361.540955:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575361.540958:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7112 00000100:00080000:1.0:1788575361.540960:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575361.540962:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575361.540969:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575361.540977:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864303872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575361.540984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575361.540986:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575361.540987:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575361.540989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7112 00000100:00080000:1.0:1788575361.540990:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575361.540992:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575361.540995:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575361.541000:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864304000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575361.541007:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575361.541009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575361.541010:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575361.541012:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7112 00000100:00080000:1.0:1788575361.541014:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575361.541016:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575361.541020:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575361.541026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864304128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575361.541032:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864303744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575361.541036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575361.541039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575361.541040:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575361.541044:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7112 00000100:00080000:1.0:1788575361.541046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575361.541048:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575361.541051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575361.541059:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864304256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575361.541091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575361.541140:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff92fc6728b100 x1875449864303744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575361.541161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864303744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575361.541197:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575361.541201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575361.541204:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864303744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575361.541210:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575361.541229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575361.541239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff92fc67289880 x1875449864303872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575361.541247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864303872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575361.541253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575361.541255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575361.541258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864303872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575361.541261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575361.541271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575361.541277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff92fc6728ad80 x1875449864304000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575361.541284:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864304000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575361.541289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575361.541291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575361.541293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575361.541312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575361.541327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575361.541337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff92fc6728b800 x1875449864304128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575361.541349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864304128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575361.541358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575361.541361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575361.541366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575361.541372:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575361.541390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575361.541399:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575361/real 1788575361] req@ffff92fc67289f80 x1875449864304256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575361.541406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864304256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575361.541411:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575361.541413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575361.541415:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575361.988887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575362.052795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575363.012830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575363.076856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575364.036914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575364.100792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575365.060877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575365.124822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575366.084873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575366.148841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575366.660852:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575366.660863:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575366.660868:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575366.660879:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7117 00000100:00080000:3.0:1788575366.660890:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575366.660898:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575366.660930:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864304384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575366.660952:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575366.660955:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575366.660956:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575366.660959:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7117 00000100:00080000:3.0:1788575366.660962:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575366.660964:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575366.660971:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575366.660982:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864304512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575366.661001:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864304384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575366.661004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575366.661009:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575366.661011:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575366.661014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7117 00000100:00080000:3.0:1788575366.661017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575366.661020:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575366.661023:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575366.661032:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864304640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575366.661038:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575366.661041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575366.661042:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575366.661050:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7117 00000100:00080000:3.0:1788575366.661053:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575366.661055:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788575366.661060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788575366.661061:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575366.661072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864304768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575366.661079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575366.661082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575366.661083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575366.661085:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7117 00000100:00080000:3.0:1788575366.661088:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575366.661091:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575366.661107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575366.661119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff92fc67339500 x1875449864304384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575366.661119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864304896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575366.661167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864304384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575366.661175:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575366.661179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575366.661187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864304384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575366.661207:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575366.661225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575366.661234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff92fc6733a300 x1875449864304512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575366.661242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864304512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575366.661248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575366.661250:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575366.661253:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575366.661257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575366.661268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575366.661274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff92fc67338000 x1875449864304640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575366.661281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864304640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575366.661291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575366.661293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575366.661295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575366.661298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575366.661309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575366.661314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff92fc67338700 x1875449864304768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575366.661322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864304768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575366.661327:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575366.661329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575366.661331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575366.661334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575366.661344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575366.661350:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff92fc6733bb80 x1875449864304896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575366.661357:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864304896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575366.661362:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575366.661363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575366.661366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864304896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575367.108884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575367.172883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575368.132938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575368.196914:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575369.156920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575369.220819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575370.180925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575370.244897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575371.204865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575371.268853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575371.780859:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575371.780870:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575371.780874:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575371.780884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7122 00000100:00080000:1.0:1788575371.780896:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575371.780904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575371.780953:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864305024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575371.780982:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575371.780986:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575371.780993:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575371.780998:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7122 00000100:00080000:1.0:1788575371.781002:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575371.781006:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575371.781032:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575371.781047:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864305152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575371.781058:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864305024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575371.781062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575371.781066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575371.781068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575371.781072:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7122 00000100:00080000:1.0:1788575371.781076:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575371.781079:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575371.781083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575371.781102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864305280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575371.781113:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575371.781116:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788575371.781120:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575371.781121:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575371.781130:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7122 00000100:00080000:1.0:1788575371.781135:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575371.781140:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575371.781147:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575371.781160:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864305408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575371.781172:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff92fc6728a680 x1875449864305024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.781178:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575371.781187:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575371.781191:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575371.781198:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7122 00000100:00080000:1.0:1788575371.781203:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575371.781207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864305024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.781211:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575371.781220:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575371.781227:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575371.781234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575371.781238:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864305536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575371.781244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864305024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575371.781278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575371.781299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575371.781309:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff92fc67289c00 x1875449864305152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575371.781317:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864305152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575371.781324:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575371.781326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575371.781328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575371.781332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575371.781343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575371.781349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff92fc67288380 x1875449864305280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575371.781356:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864305280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575371.781361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575371.781363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575371.781365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575371.781368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575371.781379:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575371.781385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff92fc67288a80 x1875449864305408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575371.781392:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864305408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575371.781397:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575371.781399:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575371.781401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575371.781405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575371.781423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575371.781433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff92fc67288700 x1875449864305536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575371.781446:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864305536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575371.781454:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575371.781457:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575371.781460:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575372.228849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575372.292824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575373.252853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575373.316804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575374.276871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575374.340837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575375.300906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575375.364884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575376.324853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575376.388849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575376.900849:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.900861:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575376.900865:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.900873:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7127 00000100:00080000:3.0:1788575376.900882:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.900887:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575376.900919:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864305664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.900965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.900969:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575376.900971:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.900975:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7127 00000100:00080000:3.0:1788575376.900978:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.900982:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575376.900991:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575376.901002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864305792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575376.901010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864305664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575376.901014:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.901022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575376.901023:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.901026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7127 00000100:00080000:3.0:1788575376.901029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.901031:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575376.901034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575376.901043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864305920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.901050:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.901053:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575376.901054:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.901056:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7127 00000100:00080000:3.0:1788575376.901060:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.901063:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575376.901066:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575376.901074:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864306048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.901080:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.901083:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788575376.901085:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575376.901086:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.901093:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7127 00000100:00080000:3.0:1788575376.901096:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.901098:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575376.901103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575376.901110:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864306176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575376.901130:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff92fc67338e00 x1875449864305664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575376.901153:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864305664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575376.901165:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575376.901170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575376.901174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864305664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575376.901198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575376.901222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575376.901235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff92fc6733a680 x1875449864305792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575376.901247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864305792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575376.901255:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575376.901258:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575376.901261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575376.901267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575376.901285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575376.901295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff92fc6733b100 x1875449864305920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575376.901307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864305920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575376.901321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575376.901324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575376.901327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864305920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575376.901333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575376.901352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575376.901360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff92fc6733bb80 x1875449864306048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575376.901368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864306048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575376.901373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575376.901375:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575376.901378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575376.901381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575376.901392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575376.901397:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff92fc67338700 x1875449864306176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575376.901404:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864306176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575376.901409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575376.901410:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575376.901413:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575377.348887:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575377.412829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575378.372854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575378.436821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575379.396844:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575379.460788:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575380.420856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575380.484820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575381.444912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575381.508780:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575382.020899:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575382.020907:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575382.020910:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575382.020919:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7132 00000100:00080000:1.0:1788575382.020928:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575382.020934:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575382.020966:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864306304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575382.020990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575382.020993:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575382.020995:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575382.020997:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7132 00000100:00080000:1.0:1788575382.020999:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575382.021001:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575382.021009:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575382.021017:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864306432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575382.021024:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575382.021027:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575382.021028:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575382.021030:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7132 00000100:00080000:1.0:1788575382.021033:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575382.021037:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864306304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575382.021038:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575382.021045:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575382.021052:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864306560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575382.021059:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575382.021062:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575382.021063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575382.021066:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7132 00000100:00080000:1.0:1788575382.021068:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575382.021070:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575382.021074:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575382.021082:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864306688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575382.021088:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575382.021090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575382.021092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575382.021094:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7132 00000100:00080000:1.0:1788575382.021097:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575382.021101:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575382.021104:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575382.021108:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575382.021117:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864306816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575382.021143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575382/real 1788575382] req@ffff92fc67289180 x1875449864306304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575382.021158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864306304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575382.021167:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575382.021170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575382.021181:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864306304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575382.021186:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575382.021202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575382.021211:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575382/real 1788575382] req@ffff92fc67289500 x1875449864306432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575382.021219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864306432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575382.021226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575382.021228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575382.021231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575382.021249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575382.021261:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575382.021268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575382/real 1788575382] req@ffff92fc6728bb80 x1875449864306560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575382.021275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864306560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575382.021281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575382.021283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575382.021285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575382.021288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575382.021299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575382.021304:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575382/real 1788575382] req@ffff92fc67288700 x1875449864306688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575382.021312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864306688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575382.021317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575382.021319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575382.021321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575382.021325:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575382.021335:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575382.021340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575382/real 1788575382] req@ffff92fc67288a80 x1875449864306816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575382.021349:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864306816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575382.021358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575382.021361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575382.021364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864306816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575382.468892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575382.532906:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575383.492931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575383.556892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575384.516872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575384.580859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575385.540942:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575385.604884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575386.564856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575386.628852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575387.140931:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575387.140945:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575387.140951:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575387.140966:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7137 00000100:00080000:1.0:1788575387.140980:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575387.140990:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575387.141040:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864306944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575387.141075:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575387.141080:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575387.141082:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575387.141085:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7137 00000100:00080000:1.0:1788575387.141090:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575387.141094:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575387.141105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575387.141116:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864307072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575387.141124:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575387.141127:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575387.141129:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575387.141133:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7137 00000100:00080000:1.0:1788575387.141135:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575387.141138:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575387.141143:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575387.141150:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864307200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575387.141159:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575387.141162:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575387.141164:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575387.141166:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7137 00000100:00080000:1.0:1788575387.141168:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575387.141172:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575387.141176:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575387.141180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864306944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575387.141186:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864307328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575387.141203:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575387.141206:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575387.141207:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575387.141210:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7137 00000100:00080000:1.0:1788575387.141212:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575387.141214:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575387.141217:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575387.141225:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864307456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575387.141272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575387.141321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575387/real 1788575387] req@ffff92fc67288380 x1875449864306944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575387.141343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864306944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575387.141352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575387.141355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575387.141358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864306944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575387.141364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575387.141384:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575387.141393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575387/real 1788575387] req@ffff92fc67289c00 x1875449864307072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575387.141401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864307072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575387.141407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575387.141409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575387.141411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575387.141431:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575387.141443:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575387.141449:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575387/real 1788575387] req@ffff92fc6728a680 x1875449864307200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575387.141457:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864307200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575387.141475:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575387.141478:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575387.141480:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575387.141484:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575387.141497:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575387.141504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575387/real 1788575387] req@ffff92fc67289f80 x1875449864307328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575387.141511:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864307328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575387.141516:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575387.141518:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575387.141520:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575387.141524:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575387.141541:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575387.141548:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575387/real 1788575387] req@ffff92fc6728b800 x1875449864307456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575387.141555:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864307456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575387.141560:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575387.141562:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575387.141565:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575387.588975:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575387.652902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575388.612866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575388.676841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575389.636930:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575389.701046:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575390.660916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575390.724867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575391.684899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575391.748926:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575392.260853:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575392.260865:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575392.260870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575392.260881:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7142 00000100:00080000:3.0:1788575392.260903:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575392.260913:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575392.260956:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864307584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575392.260990:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575392.260993:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575392.260995:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575392.260999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7142 00000100:00080000:3.0:1788575392.261002:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575392.261007:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575392.261025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575392.261043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864307712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575392.261053:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864307584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575392.261063:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575392.261068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575392.261070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575392.261075:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7142 00000100:00080000:3.0:1788575392.261081:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575392.261085:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575392.261091:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575392.261103:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864307840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575392.261111:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575392.261113:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575392.261114:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575392.261117:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7142 00000100:00080000:3.0:1788575392.261119:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575392.261121:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575392.261125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575392.261133:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864307968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575392.261139:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575392.261196:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575392.261204:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575392.261213:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7142 00000100:00080000:3.0:1788575392.261218:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575392.261221:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575392.261227:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575392.261237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575392.261247:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864308096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575392.261293:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575392.261331:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575392.261446:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575392.708868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575392.772872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575393.732882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575393.796881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575394.756884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575394.820811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788575395.333013:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788575395.333061:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575395.333077:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575395.333120:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575395.333130:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575395.333138:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575395.333148:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575395.333182:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff92fc6733b480 x1875449864308096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.333216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864308096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.333229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.333235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575395.333239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308096:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575395.333248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff92fc6733a300 x1875449864307712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.333261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864307712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.333270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.333274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575395.333277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575395.333284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff92fc67339500 x1875449864307840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.333297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864307840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.333308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.333312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575395.333315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575395.333323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff92fc6733aa00 x1875449864307968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.333335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864307968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.333344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.333348:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575395.333351:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864307968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575395.333359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575392/real 1788575395] req@ffff92fc67338000 x1875449864307584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575395.333373:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864307584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575395.333382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575395.333385:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575395.333389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864307584:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575395.780848:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575395.844836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575396.804884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575396.868831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575397.380887:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.380898:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575397.380901:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.380909:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7147 00000100:00080000:2.0:1788575397.380918:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.380925:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575397.380967:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864308224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575397.381004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.381008:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575397.381010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.381014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7147 00000100:00080000:2.0:1788575397.381019:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.381028:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575397.381038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575397.381051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864308352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575397.381061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.381065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575397.381066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.381070:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7147 00000100:00080000:2.0:1788575397.381074:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.381077:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575397.381081:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575397.381090:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864308480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575397.381098:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.381101:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575397.381103:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.381107:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7147 00000100:00080000:2.0:1788575397.381109:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.381112:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575397.381117:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575397.381126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864308608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575397.381149:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575397.381152:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575397.381154:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575397.381158:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7147 00000100:00080000:2.0:1788575397.381162:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575397.381166:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575397.381171:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575397.381182:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864308736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575397.381291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864308224:192.168.202.132@tcp:250:kworker.0 00000800:00000100:1.0:1788575397.381359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575397.381404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff92fc4617c380 x1875449864308224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.381426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864308224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.381437:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.381440:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575397.381444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864308224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575397.381453:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575397.381476:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575397.381489:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff92fc4617d180 x1875449864308352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.381503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864308352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.381512:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.381516:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575397.381519:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575397.381525:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575397.381548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575397.381560:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff92fc4617c000 x1875449864308480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.381573:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864308480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.381586:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.381589:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575397.381592:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575397.381597:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575397.381616:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575397.381626:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff92fc4617fb80 x1875449864308608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.381637:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864308608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.381644:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.381673:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575397.381676:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575397.381682:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575397.381703:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575397.381714:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575397/real 1788575397] req@ffff92fc4617e680 x1875449864308736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575397.381754:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864308736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575397.381763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575397.381766:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575397.381771:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575397.828953:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575397.892831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575398.852845:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575398.916804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575399.876846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575399.940829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575400.900935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575400.964810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575401.924884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575401.989025:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575402.500847:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.500857:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575402.500861:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.500878:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7153 00000100:00080000:3.0:1788575402.500888:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.500896:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575402.500931:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864308864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575402.500956:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.500960:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575402.500962:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.500965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7153 00000100:00080000:3.0:1788575402.500969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.500972:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575402.500981:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575402.500995:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864308992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575402.501006:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.501009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575402.501011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.501015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7153 00000100:00080000:3.0:1788575402.501019:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.501023:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575402.501031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575402.501039:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864308864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575402.501050:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864309120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575402.501064:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.501068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575402.501069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.501074:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7153 00000100:00080000:3.0:1788575402.501078:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.501082:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575402.501097:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575402.501112:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864309248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575402.501117:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575402.501131:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575402.501135:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575402.501137:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575402.501142:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7153 00000100:00080000:3.0:1788575402.501147:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575402.501152:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575402.501157:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575402.501168:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff92fc67338380 x1875449864308864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575402.501175:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864309376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575402.501204:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864308864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.501219:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.501224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575402.501229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864308864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575402.501238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575402.501258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575402.501267:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff92fc67338a80 x1875449864308992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.501275:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864308992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.501281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.501283:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575402.501285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864308992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575402.501308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575402.501319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575402.501326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff92fc6733b800 x1875449864309120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.501333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864309120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.501338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.501340:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575402.501342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575402.501345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575402.501354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575402.501360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff92fc67339880 x1875449864309248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.501366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864309248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.501370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.501372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575402.501374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575402.501376:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575402.501385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575402.501391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575402/real 1788575402] req@ffff92fc67339c00 x1875449864309376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575402.501397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864309376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575402.501403:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575402.501404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575402.501406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575402.948865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575403.012778:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575403.972864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575404.036822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575404.996894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575405.060857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575406.020894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575406.084841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575407.044918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575407.108786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575407.620831:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575407.620840:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575407.620845:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575407.620855:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7158 00000100:00080000:3.0:1788575407.620865:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575407.620872:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575407.620909:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864309504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575407.620939:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575407.620947:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575407.620949:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575407.620953:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7158 00000100:00080000:3.0:1788575407.620957:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575407.620961:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575407.620970:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575407.620983:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864309632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575407.620994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864309504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575407.620995:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575407.621003:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575407.621005:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575407.621009:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7158 00000100:00080000:3.0:1788575407.621015:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575407.621018:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575407.621023:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575407.621036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864309760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575407.621051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575407.621054:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575407.621056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575407.621059:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7158 00000100:00080000:3.0:1788575407.621063:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575407.621069:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575407.621071:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:3.0:1788575407.621075:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575407.621090:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864309888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575407.621098:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575407.621102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575407.621103:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575407.621107:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7158 00000100:00080000:3.0:1788575407.621111:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575407.621115:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff92fc67339f80 x1875449864309504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575407.621118:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575407.621127:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575407.621137:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864309504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575407.621143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864310016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575407.621156:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.621162:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575407.621166:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864309504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575407.621190:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575407.621208:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575407.621216:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff92fc6733ad80 x1875449864309632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.621223:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864309632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.621229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.621231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575407.621233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575407.621239:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575407.621255:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575407.621265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff92fc67339180 x1875449864309760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.621277:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864309760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.621284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.621287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575407.621290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575407.621302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575407.621322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575407.621333:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff92fc67339c00 x1875449864309888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.621343:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864309888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.621349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.621351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575407.621352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864309888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575407.621356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575407.621370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575407.621376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575407] req@ffff92fc67339880 x1875449864310016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575407.621383:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864310016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575407.621387:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575407.621388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575407.621390:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575408.068826:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575408.132809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575409.092852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575409.156840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575410.116797:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575410.180818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575411.140870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575411.204796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575412.164885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575412.228814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575412.740862:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575412.740874:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575412.740879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575412.740891:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7163 00000100:00080000:3.0:1788575412.740904:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575412.740912:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575412.740961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864310144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575412.741002:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575412.741006:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575412.741007:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575412.741011:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7163 00000100:00080000:3.0:1788575412.741014:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575412.741018:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575412.741031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575412.741045:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864310272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575412.741055:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864310144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575412.741060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575412.741066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575412.741068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575412.741073:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7163 00000100:00080000:3.0:1788575412.741077:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575412.741082:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575412.741087:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575412.741098:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864310400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575412.741108:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575412.741112:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575412.741114:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575412.741118:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7163 00000100:00080000:3.0:1788575412.741122:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575412.741126:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575412.741131:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575412.741135:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575412.741145:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864310528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575412.741153:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575412.741156:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575412.741157:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575412.741160:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7163 00000100:00080000:3.0:1788575412.741162:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575412.741165:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575412.741168:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575412.741177:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864310656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575412.741183:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff92fc6733b800 x1875449864310144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.741200:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864310144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.741208:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.741211:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575412.741215:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864310144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575412.741236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575412.741251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575412.741258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff92fc67338a80 x1875449864310272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.741265:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864310272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.741275:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.741278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575412.741285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575412.741289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575412.741300:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575412.741305:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff92fc67338380 x1875449864310400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.741312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864310400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.741317:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.741319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575412.741320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575412.741323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575412.741333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575412.741338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff92fc67338000 x1875449864310528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.741344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864310528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.741349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.741351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575412.741353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575412.741356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575412.741366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575412.741371:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff92fc6733aa00 x1875449864310656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575412.741377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864310656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575412.741381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575412.741383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575412.741385:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575413.188858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575413.252856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575414.212867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575414.276800:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575415.236852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575415.300853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575416.260841:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575416.324812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575417.284915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575417.348829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575417.860821:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575417.860829:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575417.860833:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575417.860842:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7168 00000100:00080000:3.0:1788575417.860851:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575417.860857:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575417.860888:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864310784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575417.860911:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575417.860914:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575417.860915:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575417.860917:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7168 00000100:00080000:3.0:1788575417.860920:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575417.860922:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575417.860930:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575417.860940:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864310912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575417.860948:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575417.860952:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575417.860953:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575417.860956:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7168 00000100:00080000:3.0:1788575417.860958:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575417.860961:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864310784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575417.860962:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575417.860969:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575417.860976:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864311040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575417.860981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575417.860983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575417.860985:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575417.860987:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7168 00000100:00080000:3.0:1788575417.860990:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575417.860992:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575417.860996:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575417.861004:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864311168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575417.861011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575417.861013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575417.861014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575417.861016:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7168 00000100:00080000:3.0:1788575417.861019:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575417.861023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575417.861024:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575417.861030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575417.861036:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864311296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575417.861072:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff92fc67339500 x1875449864310784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.861093:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864310784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.861101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.861103:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575417.861106:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864310784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575417.861128:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575417.861152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575417.861167:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff92fc6733a300 x1875449864310912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.861180:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864310912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.861186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.861188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575417.861191:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864310912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575417.861195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575417.861210:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575417.861217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff92fc6733b480 x1875449864311040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.861225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864311040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.861230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.861233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575417.861235:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575417.861238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575417.861249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575417.861255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff92fc67338700 x1875449864311168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.861262:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864311168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.861268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.861270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575417.861272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575417.861275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575417.861285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575417.861299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff92fc6733bb80 x1875449864311296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.861307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864311296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.861312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.861314:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575417.861316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575418.308861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575418.372816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575419.332898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575419.396880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575420.356880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575420.420809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575421.380836:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575421.444840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575422.404901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575422.468891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575422.980847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575422.980862:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575422.980868:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575422.980880:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7173 00000100:00080000:3.0:1788575422.980894:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575422.980904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575422.980947:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864311424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575422.980973:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575422.980978:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575422.980979:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575422.980982:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7173 00000100:00080000:3.0:1788575422.980985:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575422.980987:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575422.980997:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575422.981006:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864311552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575422.981013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575422.981015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575422.981016:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575422.981019:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7173 00000100:00080000:3.0:1788575422.981020:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575422.981022:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575422.981025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575422.981032:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864311424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575422.981033:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864311680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575422.981045:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575422.981047:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575422.981047:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575422.981050:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7173 00000100:00080000:3.0:1788575422.981052:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575422.981054:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575422.981058:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575422.981067:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864311808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575422.981072:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575422.981074:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575422.981076:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575422.981078:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7173 00000100:00080000:3.0:1788575422.981087:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575422.981092:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575422.981095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575422.981104:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864311936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575422.981111:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575422.981156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff92fc6733b100 x1875449864311424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.981172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864311424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.981185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.981188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575422.981191:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864311424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575422.981197:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575422.981212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575422.981220:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff92fc6733a680 x1875449864311552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.981227:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864311552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.981233:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.981235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575422.981237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575422.981259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575422.981269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575422.981275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff92fc67338e00 x1875449864311680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.981282:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864311680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.981287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.981289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575422.981291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575422.981293:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575422.981303:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575422.981308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff92fc6733bb80 x1875449864311808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.981316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864311808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.981320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.981322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575422.981323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575422.981326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575422.981336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575422.981341:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff92fc67338700 x1875449864311936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575422.981348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864311936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575422.981353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575422.981354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575422.981356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864311936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575423.428883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575423.492875:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575424.452901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575424.516860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575425.476841:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575425.540840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575426.500896:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575426.564822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575427.524824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575427.588822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575428.100834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575428.100845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575428.100849:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575428.100860:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7178 00000100:00080000:3.0:1788575428.100871:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575428.100877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575428.100908:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864312064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575428.100931:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575428.100935:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575428.100937:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575428.100941:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7178 00000100:00080000:3.0:1788575428.100945:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575428.100950:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575428.100961:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575428.100974:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864312192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575428.100982:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864312064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575428.100984:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575428.100987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575428.100989:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575428.100993:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7178 00000100:00080000:3.0:1788575428.100996:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575428.100999:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575428.101012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575428.101026:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864312320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575428.101035:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575428.101038:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788575428.101041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575428.101041:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575428.101044:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7178 00000100:00080000:3.0:1788575428.101047:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575428.101051:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575428.101056:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575428.101065:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864312448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575428.101079:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575428.101082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575428.101083:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575428.101087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7178 00000100:00100000:1.0:1788575428.101089:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575428/real 1788575428] req@ffff92fc6733b480 x1875449864312064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575428.101090:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575428.101094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575428.101098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575428.101108:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864312064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575428.101108:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864312576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575428.101125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575428.101128:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575428.101132:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864312064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575428.101154:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575428.101178:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575428.101191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575428/real 1788575428] req@ffff92fc6733a300 x1875449864312192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575428.101203:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864312192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575428.101212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575428.101215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575428.101218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575428.101225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575428.101245:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575428.101255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575428/real 1788575428] req@ffff92fc67339500 x1875449864312320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575428.101268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864312320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575428.101277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575428.101281:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575428.101285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575428.101292:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575428.101317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575428.101330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575428/real 1788575428] req@ffff92fc6733aa00 x1875449864312448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575428.101346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864312448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575428.101354:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575428.101358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575428.101375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575428.101381:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575428.101403:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575428.101415:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575428/real 1788575428] req@ffff92fc67338000 x1875449864312576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575428.101427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864312576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575428.101435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575428.101438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575428.101442:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575428.548880:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575428.612847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575429.572821:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575429.636834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575430.596845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575430.660834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575431.620857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575431.684821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575432.644861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575432.708786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575433.220876:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575433.220883:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575433.220894:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575433.220903:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7183 00000100:00080000:0.0:1788575433.220912:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575433.220918:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575433.220948:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864312704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575433.220974:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575433.220977:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575433.220978:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575433.220981:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7183 00000100:00080000:0.0:1788575433.220983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575433.220990:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575433.220998:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575433.221006:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864312832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575433.221014:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575433.221016:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575433.221018:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575433.221021:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7183 00000100:00080000:0.0:1788575433.221023:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575433.221026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575433.221030:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864312704:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788575433.221031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575433.221042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864312960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575433.221049:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575433.221052:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575433.221053:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575433.221056:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7183 00000100:00080000:0.0:1788575433.221058:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575433.221060:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575433.221063:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575433.221070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864313088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575433.221076:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575433.221078:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575433.221080:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575433.221082:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7183 00000100:00080000:0.0:1788575433.221084:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575433.221091:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575433.221091:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788575433.221097:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575433.221108:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864313216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575433.221137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff92fc67048000 x1875449864312704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.221152:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864312704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.221160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.221163:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575433.221166:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864312704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575433.221172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575433.221189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575433.221197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff92fc67049f80 x1875449864312832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.221206:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864312832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.221212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.221214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575433.221216:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575433.221235:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575433.221246:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575433.221253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff92fc67048700 x1875449864312960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.221260:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864312960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.221273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.221276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575433.221280:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864312960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575433.221286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575433.221306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575433.221334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff92fc67048380 x1875449864313088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.221342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864313088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.221348:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.221351:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575433.221354:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575433.221361:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575433.221383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575433.221395:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575433/real 1788575433] req@ffff92fc6704a300 x1875449864313216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575433.221416:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864313216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575433.221422:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575433.221424:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575433.221427:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575433.668910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575433.732816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575434.692937:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575434.756871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575435.716818:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575435.780822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575436.740825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575436.804826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575437.764868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575437.828835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575438.340818:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575438.340826:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575438.340830:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575438.340845:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7188 00000100:00080000:0.0:1788575438.340856:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575438.340863:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575438.340894:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864313344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575438.340920:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575438.340924:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575438.340925:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575438.340928:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7188 00000100:00080000:0.0:1788575438.340930:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575438.340932:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575438.340940:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575438.340951:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864313472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575438.340959:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575438.340963:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575438.340963:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864313344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575438.340966:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575438.340973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7188 00000100:00080000:0.0:1788575438.340975:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575438.340977:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575438.340980:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575438.340988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864313600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575438.340994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575438.340996:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575438.340997:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575438.341000:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7188 00000100:00080000:0.0:1788575438.341003:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575438.341005:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575438.341008:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575438.341015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864313728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575438.341021:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575438.341023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575438.341024:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575438.341029:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7188 00000800:00000100:1.0:1788575438.341031:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575438.341033:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575438.341036:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575438.341040:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575438.341047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864313856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575438.341070:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff92fc67048a80 x1875449864313344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575438.341083:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864313344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575438.341091:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575438.341094:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575438.341097:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864313344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575438.341114:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575438.341129:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575438.341137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff92fc6704a300 x1875449864313472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575438.341145:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864313472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575438.341151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575438.341153:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575438.341155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575438.341159:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575438.341169:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575438.341175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff92fc67048380 x1875449864313600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575438.341182:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864313600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575438.341188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575438.341190:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575438.341192:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575438.341195:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575438.341206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575438.341212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff92fc67048700 x1875449864313728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575438.341219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864313728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575438.341224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575438.341226:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575438.341228:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575438.341231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575438.341240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575438.341246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575438/real 1788575438] req@ffff92fc67049f80 x1875449864313856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575438.341253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864313856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575438.341258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575438.341260:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575438.341262:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864313856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575438.788889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575438.852847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575439.812851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575439.876798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575440.836889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575440.900882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575441.860860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575441.924811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575442.884883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575442.948834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575443.460820:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575443.460827:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575443.460830:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575443.460837:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7194 00000100:00080000:0.0:1788575443.460845:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575443.460851:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575443.460882:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864313984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575443.460909:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575443.460913:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575443.460914:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575443.460918:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7194 00000100:00080000:0.0:1788575443.460922:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575443.460925:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575443.460934:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575443.460946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864314112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575443.460957:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575443.460961:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864313984:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575443.460962:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575443.460966:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575443.460973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7194 00000100:00080000:0.0:1788575443.460977:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575443.460980:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575443.460986:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575443.460995:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864314240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575443.461001:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575443.461003:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575443.461004:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575443.461007:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7194 00000100:00080000:0.0:1788575443.461017:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575443.461020:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575443.461022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575443.461028:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575443.461031:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864314368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575443.461038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575443.461040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575443.461041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575443.461043:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7194 00000100:00080000:0.0:1788575443.461045:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575443.461047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575443.461050:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575443.461057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864314496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575443.461077:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff92fc67048000 x1875449864313984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575443.461095:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864313984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575443.461104:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575443.461108:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575443.461112:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864313984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575443.461132:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575443.461152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575443.461164:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff92fc6704aa00 x1875449864314112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575443.461174:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864314112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575443.461183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575443.461186:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575443.461189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575443.461196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575443.461213:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575443.461254:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff92fc6704a680 x1875449864314240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575443.461266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864314240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575443.461273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575443.461276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575443.461279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575443.461284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575443.461302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575443.461312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff92fc6704ad80 x1875449864314368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575443.461322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864314368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575443.461329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575443.461332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575443.461335:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575443.461341:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575443.461358:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575443.461368:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575443/real 1788575443] req@ffff92fc67049880 x1875449864314496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575443.461380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864314496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575443.461388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575443.461390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575443.461393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575443.908871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575443.972827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575444.932838:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575444.996826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575445.956838:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575446.020849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575446.980816:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575447.044834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575448.004935:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575448.068807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575448.580792:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575448.580801:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575448.580806:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575448.580816:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7199 00000100:00080000:3.0:1788575448.580826:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575448.580834:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575448.580872:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864314624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575448.580897:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575448.580902:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575448.580903:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575448.580908:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7199 00000100:00080000:3.0:1788575448.580919:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575448.580924:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575448.580936:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575448.580951:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864314752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575448.580957:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864314624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575448.580961:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575448.580966:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575448.580968:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575448.580971:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7199 00000100:00080000:3.0:1788575448.580974:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575448.580976:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575448.580979:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575448.580988:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864314880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575448.580994:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575448.580998:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575448.581000:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575448.581004:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7199 00000100:00080000:3.0:1788575448.581007:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575448.581011:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575448.581017:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575448.581028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864315008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575448.581037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575448.581041:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575448.581043:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575448.581048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575448.581052:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7199 00000100:00080000:3.0:1788575448.581058:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575448.581062:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575448.581068:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575448.581079:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864315136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575448.581106:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff92fc67338380 x1875449864314624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575448.581128:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864314624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575448.581139:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575448.581143:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575448.581147:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864314624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575448.581211:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575448.581236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575448.581249:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff92fc67338a80 x1875449864314752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575448.581261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864314752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575448.581276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575448.581279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575448.581282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575448.581288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575448.581307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575448.581318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff92fc6733b800 x1875449864314880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575448.581329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864314880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575448.581336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575448.581339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575448.581342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864314880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575448.581347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575448.581365:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575448.581375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff92fc67339880 x1875449864315008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575448.581386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864315008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575448.581393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575448.581396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575448.581400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575448.581405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575448.581424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575448.581436:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff92fc67339c00 x1875449864315136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575448.581447:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864315136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575448.581455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575448.581458:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575448.581462:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575449.028885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575449.092845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575450.052884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575450.116854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575451.076954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575451.140883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575452.100941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575452.164891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575453.124915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575453.188849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575453.700897:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.700910:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575453.700915:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.700928:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7204 00000100:00080000:0.0:1788575453.700941:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.700949:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575453.700999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864315264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575453.701030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.701034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575453.701036:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.701041:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7204 00000100:00080000:0.0:1788575453.701044:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.701049:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575453.701061:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575453.701074:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864315392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575453.701077:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864315264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575453.701092:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.701096:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575453.701098:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.701102:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7204 00000100:00080000:0.0:1788575453.701106:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.701110:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575453.701117:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575453.701140:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864315520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575453.701150:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.701154:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575453.701156:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.701160:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7204 00000800:00000100:1.0:1788575453.701164:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575453.701166:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.701173:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575453.701179:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575453.701190:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864315648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575453.701198:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575453.701200:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575453.701202:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575453.701204:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7204 00000100:00080000:0.0:1788575453.701207:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575453.701210:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575453.701215:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575453.701217:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff92fc67049c00 x1875449864315264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575453.701227:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864315776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575453.701235:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864315264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.701244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.701248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575453.701251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864315264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575453.701277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575453.701299:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575453.701312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff92fc67049180 x1875449864315392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.701324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864315392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.701329:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.701331:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575453.701334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575453.701338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575453.701352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575453.701359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff92fc6704bb80 x1875449864315520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.701366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864315520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.701371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.701373:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575453.701375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575453.701378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575453.701388:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575453.701394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff92fc67049500 x1875449864315648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.701400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864315648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.701406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.701408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575453.701410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575453.701413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575453.701422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575453.701427:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff92fc6704b100 x1875449864315776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.701432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864315776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.701437:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.701438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575453.701440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864315776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575454.148918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575454.212917:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575455.172889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575455.236803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575456.196894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575456.260832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575457.220906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575457.284802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575458.244892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575458.308880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575458.820883:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575458.820893:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575458.820897:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575458.820908:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7209 00000100:00080000:0.0:1788575458.820920:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575458.820928:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575458.820968:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864315904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575458.820988:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575458.820991:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575458.820993:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575458.820995:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7209 00000100:00080000:0.0:1788575458.820998:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575458.821000:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575458.821010:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575458.821021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864316032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575458.821027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864315904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575458.821030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575458.821035:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575458.821038:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575458.821041:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7209 00000100:00080000:0.0:1788575458.821043:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575458.821045:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575458.821048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575458.821056:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864316160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575458.821063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575458.821065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575458.821066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575458.821068:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7209 00000100:00080000:0.0:1788575458.821070:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575458.821072:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575458.821075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575458.821082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864316288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575458.821087:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575458.821089:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575458.821090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575458.821092:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7209 00000100:00080000:0.0:1788575458.821123:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575458.821126:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575458.821129:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575458.821138:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864316416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575458.821180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575458.821213:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575458.821233:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575458.821256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575459.268891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575459.332798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575460.292895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575460.356872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575461.316883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575461.380899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788575461.893033:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:3.0:1788575461.893088:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575461.893105:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575461.893151:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575461.893163:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575461.893172:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575461.893178:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575461.893223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff92fc6704b100 x1875449864316032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575461.893264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864316032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575461.893277:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575461.893282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575461.893287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575461.893299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff92fc67049500 x1875449864316160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575461.893315:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864316160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575461.893322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575461.893327:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575461.893330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575461.893337:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff92fc6704bb80 x1875449864316288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575461.893348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864316288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575461.893355:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575461.893359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575461.893362:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575461.893369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff92fc67049180 x1875449864316416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575461.893381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864316416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575461.893388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575461.893391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575461.893394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316416:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575461.893401:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575461] req@ffff92fc67048e00 x1875449864315904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575461.893412:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864315904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575461.893419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575461.893422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575461.893425:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864315904:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575462.340904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575462.404857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575463.364951:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575463.428878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575463.940870:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.940883:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575463.940888:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.940899:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7214 00000100:00080000:2.0:1788575463.940918:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.940928:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575463.940976:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864316544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.941013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.941018:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.941020:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.941024:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7214 00000100:00080000:2.0:1788575463.941027:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.941032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575463.941044:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.941060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864316672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575463.941063:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864316544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575463.941095:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.941102:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575463.941104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.941109:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7214 00000100:00080000:2.0:1788575463.941113:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.941117:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575463.941122:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.941134:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864316800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.941144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.941146:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.941148:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.941152:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7214 00000100:00080000:2.0:1788575463.941156:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.941160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575463.941165:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575463.941175:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864316928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.941183:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575463.941189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575463.941189:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575463.941192:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.941198:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7214 00000100:00080000:2.0:1788575463.941202:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.941206:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575463.941213:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575463.941224:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864317056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575463.941261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff92fc4617f800 x1875449864316544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.941284:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864316544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.941297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.941301:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575463.941305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864316544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575463.941339:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575463.941363:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575463.941376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff92fc4617f100 x1875449864316672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.941389:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864316672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.941398:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.941402:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575463.941405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575463.941416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575463.941440:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575463.941453:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff92fc4617dc00 x1875449864316800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.941465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864316800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.941473:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.941477:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575463.941480:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575463.941485:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575463.941507:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575463.941517:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff92fc4617c700 x1875449864316928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.941528:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864316928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.941536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.941539:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575463.941542:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864316928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575463.941548:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575463.941568:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575463.941579:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff92fc4617ca80 x1875449864317056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575463.941590:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864317056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575463.941597:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575463.941600:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575463.941603:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575464.388882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575464.452868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575465.412915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575465.476872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575466.436848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575466.500895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575467.460861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575467.524874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575468.484867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575468.548821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575469.060869:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575469.060881:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575469.060885:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575469.060896:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7219 00000100:00080000:0.0:1788575469.060908:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575469.060915:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575469.060953:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864317184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575469.060981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575469.060984:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575469.060986:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575469.060989:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7219 00000100:00080000:0.0:1788575469.060993:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575469.060996:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575469.061005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575469.061014:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864317312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575469.061022:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575469.061026:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575469.061027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864317184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575469.061028:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575469.061037:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7219 00000100:00080000:0.0:1788575469.061042:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575469.061046:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575469.061051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575469.061065:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864317440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575469.061076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575469.061080:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575469.061082:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575469.061086:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7219 00000100:00080000:0.0:1788575469.061090:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575469.061094:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575469.061100:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788575469.061103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575469.061116:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864317568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575469.061123:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575469.061125:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575469.061127:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575469.061129:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7219 00000100:00080000:0.0:1788575469.061132:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575469.061135:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575469.061139:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575469.061149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864317696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575469.061159:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575469/real 1788575469] req@ffff92fc67049c00 x1875449864317184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575469.061179:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864317184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575469.061188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575469.061192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575469.061195:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864317184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575469.061219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575469.061237:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575469.061247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575469/real 1788575469] req@ffff92fc67049880 x1875449864317312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575469.061255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864317312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575469.061261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575469.061264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575469.061266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575469.061270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575469.061295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575469.061302:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575469/real 1788575469] req@ffff92fc6704ad80 x1875449864317440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575469.061310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864317440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575469.061315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575469.061318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575469.061320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575469.061323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575469.061334:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575469.061340:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575469/real 1788575469] req@ffff92fc6704a680 x1875449864317568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575469.061348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864317568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575469.061353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575469.061355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575469.061357:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575469.061362:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575469.061381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575469.061428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575469/real 1788575469] req@ffff92fc6704aa00 x1875449864317696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575469.061436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864317696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575469.061442:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575469.061444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575469.061446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575469.508874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575469.572827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575470.532894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575470.596819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575471.556872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575471.620847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575472.580915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575472.644818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575473.604873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575473.668815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575474.180879:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575474.180888:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575474.180891:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575474.180901:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7224 00000100:00080000:1.0:1788575474.180912:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575474.180918:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575474.180954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864317824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575474.180984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575474.180987:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575474.180988:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575474.180991:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7224 00000100:00080000:1.0:1788575474.180994:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575474.180998:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575474.181006:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575474.181015:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864317952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575474.181025:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575474.181034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575474.181035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864317824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575474.181036:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575474.181051:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7224 00000100:00080000:1.0:1788575474.181054:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575474.181057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575474.181060:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575474.181071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864318080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575474.181077:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575474.181080:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575474.181081:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575474.181084:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7224 00000100:00080000:1.0:1788575474.181087:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575474.181090:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575474.181094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575474.181101:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864318208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575474.181107:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575474.181109:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575474.181111:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575474.181115:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7224 00000100:00080000:1.0:1788575474.181120:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575474.181124:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788575474.181131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575474.181134:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575474.181152:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864318336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575474.181199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff92fc6728ad80 x1875449864317824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.181219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864317824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.181229:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.181233:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575474.181236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864317824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575474.181266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575474.181294:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575474.181309:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff92fc67289880 x1875449864317952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.181320:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864317952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.181326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.181329:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575474.181331:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864317952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575474.181337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575474.181353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575474.181360:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff92fc6728b100 x1875449864318080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.181368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864318080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.181374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.181376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575474.181378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575474.181400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575474.181415:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575474.181422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff92fc67288000 x1875449864318208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.181430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864318208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.181435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.181438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575474.181440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575474.181444:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575474.181471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575474.181478:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575474/real 1788575474] req@ffff92fc67288e00 x1875449864318336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575474.181484:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864318336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575474.181490:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575474.181494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575474.181497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575474.628914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575474.692803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575475.652898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575475.716832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575476.676873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575476.740820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575477.700847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575477.764808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575478.724837:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575478.788796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575479.300858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575479.300866:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575479.300870:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575479.300878:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7229 00000100:00080000:3.0:1788575479.300887:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575479.300893:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575479.300930:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864318464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575479.300953:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575479.300956:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575479.300958:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575479.300962:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7229 00000100:00080000:3.0:1788575479.300967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575479.300972:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575479.300982:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575479.300995:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864318464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575479.300996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864318592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575479.301011:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575479.301013:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575479.301015:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575479.301018:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7229 00000100:00080000:3.0:1788575479.301021:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575479.301026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575479.301031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575479.301041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864318720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575479.301048:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575479.301050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575479.301051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575479.301054:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7229 00000100:00080000:3.0:1788575479.301057:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575479.301059:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575479.301062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575479.301068:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575479.301075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864318848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575479.301083:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575479.301086:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575479.301087:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575479.301090:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7229 00000100:00080000:3.0:1788575479.301092:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575479.301095:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575479.301098:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575479.301106:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864318976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575479.301118:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff92fc67339180 x1875449864318464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.301138:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864318464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.301147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.301150:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575479.301153:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864318464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575479.301172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575479.301190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575479.301198:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff92fc6733ad80 x1875449864318592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.301206:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864318592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.301212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.301215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575479.301217:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575479.301221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575479.301232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575479.301239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff92fc67339f80 x1875449864318720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.301246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864318720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.301251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.301253:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575479.301256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575479.301259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575479.301269:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575479.301275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff92fc67339c00 x1875449864318848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.301281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864318848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.301286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.301288:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575479.301290:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575479.301294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575479.301303:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575479.301308:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575479/real 1788575479] req@ffff92fc67339880 x1875449864318976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575479.301316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864318976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575479.301321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575479.301322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575479.301324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864318976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575479.748863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575479.812853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575480.772894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575480.836837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575481.796864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575481.860890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575482.820874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575482.884846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575483.844842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575483.908872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575484.420912:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575484.420925:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575484.420930:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575484.420942:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7235 00000100:00080000:3.0:1788575484.420955:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575484.420963:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575484.421013:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864319104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575484.421056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575484.421061:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575484.421063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575484.421068:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7235 00000100:00080000:3.0:1788575484.421072:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575484.421079:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575484.421092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575484.421099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864319104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575484.421109:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864319232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575484.421122:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575484.421126:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575484.421128:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575484.421132:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7235 00000100:00080000:3.0:1788575484.421136:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575484.421140:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575484.421146:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575484.421159:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864319360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575484.421170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575484.421172:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575484.421177:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575484.421179:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575484.421185:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7235 00000100:00080000:3.0:1788575484.421190:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575484.421194:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575484.421200:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575484.421214:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864319488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575484.421223:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff92fc6733b800 x1875449864319104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575484.421226:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575484.421236:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575484.421239:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575484.421244:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7235 00000100:00100000:0.0:1788575484.421247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864319104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575484.421250:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575484.421260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575484.421260:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:0.0:1788575484.421267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000008:00100000:3.0:1788575484.421269:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575484.421270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864319104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575484.421286:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864319616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575484.421297:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575484.421313:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575484.421323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff92fc67338a80 x1875449864319232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.421331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864319232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.421337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.421339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575484.421341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575484.421344:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575484.421353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575484.421358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff92fc67338380 x1875449864319360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.421365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864319360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.421369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.421371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575484.421372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575484.421375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575484.421385:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575484.421390:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff92fc67338000 x1875449864319488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.421398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864319488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.421402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.421404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575484.421405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575484.421413:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575484.421422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575484.421427:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575484/real 1788575484] req@ffff92fc6733aa00 x1875449864319616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575484.421434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864319616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575484.421438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575484.421440:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575484.421443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575484.868968:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575484.932867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575485.892820:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575485.956821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575486.916881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575486.980856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575487.940836:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575488.004842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575488.964872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575489.028829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575489.540847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575489.540854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575489.540857:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575489.540865:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7240 00000100:00080000:2.0:1788575489.540874:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575489.540881:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575489.540916:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864319744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575489.540946:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575489.540950:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575489.540951:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575489.540962:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7240 00000100:00080000:2.0:1788575489.540968:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575489.540971:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575489.540989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575489.541000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864319872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575489.541009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575489.541011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575489.541013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575489.541025:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7240 00000100:00080000:2.0:1788575489.541029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575489.541032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575489.541036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575489.541047:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864320000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575489.541055:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575489.541064:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575489.541066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575489.541069:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7240 00000100:00080000:2.0:1788575489.541073:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575489.541076:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575489.541081:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575489.541088:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864320128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575489.541093:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575489.541095:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575489.541096:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575489.541098:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7240 00000100:00080000:2.0:1788575489.541100:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575489.541102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575489.541105:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575489.541117:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864320256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575489.541148:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864319744:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788575489.541214:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575489.541252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff92fc4617f480 x1875449864319744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.541266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864319744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.541273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.541277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575489.541281:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864319744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575489.541286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575489.541301:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575489.541309:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff92fc4617ed80 x1875449864319872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.541316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864319872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.541321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.541322:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575489.541324:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864319872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575489.541328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575489.541343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575489.541361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff92fc4617ca80 x1875449864320000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.541373:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864320000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.541380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.541383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575489.541386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575489.541391:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575489.541411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575489.541421:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff92fc4617c700 x1875449864320128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.541430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864320128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.541434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.541436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575489.541438:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575489.541442:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575489.541453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575489.541459:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575489/real 1788575489] req@ffff92fc4617dc00 x1875449864320256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575489.541465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864320256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575489.541478:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575489.541481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575489.541484:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575489.988870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575490.052843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575491.012907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575491.076859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575492.037015:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575492.100843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575493.060916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575493.124852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575494.084905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575494.148839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575494.660858:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.660868:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575494.660871:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.660881:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7245 00000100:00080000:1.0:1788575494.660891:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.660898:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575494.660932:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864320384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.660956:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.660960:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575494.660961:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.660965:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7245 00000100:00080000:1.0:1788575494.660969:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.660973:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575494.660983:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575494.660994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864320384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575494.660996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864320512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.661009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.661013:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575494.661014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.661018:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7245 00000100:00080000:1.0:1788575494.661022:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.661025:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575494.661030:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575494.661041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864320640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575494.661046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575494.661056:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575494.661059:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575494.661062:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.661066:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7245 00000100:00080000:1.0:1788575494.661075:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575494.661079:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575494.661083:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff92fc6728a300 x1875449864320384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575494.661086:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575494.661102:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864320384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575494.661105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864320768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.661120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575494.661136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.661140:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575494.661142:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864320384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575494.661149:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575494.661152:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575494.661156:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7245 00000100:00080000:1.0:1788575494.661160:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575494.661162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320512:192.168.202.132@tcp:38:kworker.0 00000100:00080000:1.0:1788575494.661167:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575494.661173:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575494.661187:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864320896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575494.661190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575494.661206:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff92fc6728b480 x1875449864320512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.661224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864320512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.661230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.661232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575494.661234:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575494.661238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575494.661251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575494.661257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff92fc6728aa00 x1875449864320640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.661264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864320640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.661268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.661270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575494.661272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575494.661275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575494.661283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575494.661288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff92fc67288e00 x1875449864320768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.661294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864320768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.661302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.661304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575494.661306:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575494.661314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575494.661323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575494.661328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff92fc67288000 x1875449864320896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575494.661334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864320896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575494.661339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575494.661340:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575494.661342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864320896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575495.108949:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575495.172905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575496.132905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575496.196892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575497.156894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575497.220867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575498.180893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575498.244832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575499.204924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575499.268859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575499.780834:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.780844:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575499.780848:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.780858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7250 00000100:00080000:1.0:1788575499.780868:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.780875:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575499.780908:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864321024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575499.780935:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.780939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575499.780941:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.780944:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7250 00000100:00080000:1.0:1788575499.780948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.780952:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575499.780960:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575499.780972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864321152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575499.780983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864321024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575499.780983:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.780991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575499.780995:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.780999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7250 00000100:00080000:1.0:1788575499.781002:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.781005:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575499.781010:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575499.781020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864321280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575499.781029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.781032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575499.781034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575499.781038:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7250 00000100:00080000:1.0:1788575499.781048:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575499.781053:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788575499.781056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575499.781061:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575499.781075:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864321408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575499.781085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575499.781088:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575499.781091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575499.781092:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff92fc6728b100 x1875449864321024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.781100:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7250 00000100:00080000:1.0:1788575499.781111:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575499.781113:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864321024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.781118:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575499.781125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575499.781132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.781139:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575499.781140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864321536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575499.781147:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864321024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575499.781173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575499.781192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575499.781202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff92fc67289880 x1875449864321152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.781215:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864321152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.781223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.781228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575499.781238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575499.781246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575499.781288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575499.781300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff92fc6728ad80 x1875449864321280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.781313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864321280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.781321:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.781325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575499.781328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575499.781336:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575499.781359:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575499.781369:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff92fc6728b800 x1875449864321408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.781381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864321408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.781391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.781395:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575499.781399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575499.781408:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575499.781433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575499.781449:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff92fc67289f80 x1875449864321536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575499.781465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864321536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575499.781470:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575499.781473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575499.781476:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575500.228872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575500.292846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575501.252855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575501.316843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575502.276886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575502.340915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575503.300872:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575503.364921:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575504.324868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575504.388846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575504.900874:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575504.900883:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575504.900887:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575504.900897:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7255 00000100:00080000:3.0:1788575504.900906:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575504.900912:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575504.900947:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864321664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575504.900972:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575504.900975:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575504.900976:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575504.900979:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7255 00000100:00080000:3.0:1788575504.900981:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575504.900984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575504.900991:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575504.901003:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864321792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575504.901011:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575504.901013:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575504.901014:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575504.901017:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7255 00000100:00080000:3.0:1788575504.901019:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575504.901021:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575504.901024:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575504.901032:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864321664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575504.901033:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864321920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575504.901045:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575504.901046:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575504.901047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575504.901049:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7255 00000100:00080000:3.0:1788575504.901051:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575504.901052:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575504.901055:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575504.901061:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864322048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575504.901067:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575504.901068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575504.901069:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575504.901071:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7255 00000100:00080000:3.0:1788575504.901073:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575504.901076:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575504.901078:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575504.901083:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864322176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575504.901113:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575504.901153:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff92fc67339500 x1875449864321664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575504.901181:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864321664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575504.901192:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575504.901195:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575504.901199:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864321664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575504.901206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575504.901229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575504.901241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff92fc6733a300 x1875449864321792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575504.901253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864321792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575504.901261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575504.901264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575504.901267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575504.901290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575504.901309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575504.901320:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff92fc6733b480 x1875449864321920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575504.901332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864321920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575504.901339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575504.901341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575504.901344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864321920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575504.901349:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575504.901367:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575504.901378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff92fc67338700 x1875449864322048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575504.901388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864322048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575504.901395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575504.901398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575504.901401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575504.901407:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575504.901460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575504.901472:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff92fc6733bb80 x1875449864322176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575504.901482:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864322176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575504.901495:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575504.901498:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575504.901501:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575505.348820:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575505.412822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575506.372853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575506.436869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575507.396850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575507.460865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575508.420859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575508.484840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575509.444855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575509.508853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575510.020843:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575510.020850:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575510.020853:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575510.020861:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7260 00000100:00080000:2.0:1788575510.020868:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575510.020875:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575510.020909:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864322304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575510.020931:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575510.020934:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575510.020936:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575510.020938:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7260 00000100:00080000:2.0:1788575510.020941:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575510.020943:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575510.020950:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575510.020959:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864322432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575510.020966:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575510.020968:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575510.020969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575510.020971:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7260 00000100:00080000:2.0:1788575510.020972:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575510.020974:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575510.020980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864322304:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788575510.020980:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575510.020989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864322560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575510.020995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575510.020997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575510.020998:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575510.021000:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7260 00000100:00080000:2.0:1788575510.021002:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575510.021003:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575510.021006:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575510.021013:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864322688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575510.021019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575510.021021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575510.021022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575510.021024:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7260 00000100:00080000:2.0:1788575510.021026:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575510.021028:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575510.021031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575510.021040:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864322816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575510.021041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575510.021083:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575510/real 1788575510] req@ffff92fc4617f100 x1875449864322304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575510.021100:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864322304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575510.021107:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575510.021109:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575510.021112:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864322304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575510.021117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575510.021132:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575510.021139:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575510/real 1788575510] req@ffff92fc4617f800 x1875449864322432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575510.021146:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864322432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575510.021152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575510.021154:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575510.021156:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575510.021173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575510.021188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575510.021197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575510/real 1788575510] req@ffff92fc4617e680 x1875449864322560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575510.021208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864322560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575510.021216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575510.021219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575510.021221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575510.021225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575510.021237:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575510.021243:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575510/real 1788575510] req@ffff92fc4617fb80 x1875449864322688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575510.021250:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864322688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575510.021254:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575510.021256:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575510.021258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575510.021262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575510.021270:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575510.021275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575510/real 1788575510] req@ffff92fc4617c000 x1875449864322816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575510.021281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864322816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575510.021285:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575510.021287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575510.021289:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864322816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575510.468882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575510.532839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575511.492871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575511.556833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575512.516863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575512.580833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575513.540897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575513.604828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575514.564871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575514.628852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575515.140831:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575515.140839:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575515.140842:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575515.140849:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7265 00000100:00080000:3.0:1788575515.140857:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575515.140862:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575515.140899:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864322944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575515.140919:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575515.140921:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575515.140923:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575515.140928:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7265 00000100:00080000:3.0:1788575515.140930:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575515.140932:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575515.140939:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575515.140947:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864323072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575515.140954:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575515.140960:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788575515.140961:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864322944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575515.140962:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575515.140969:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7265 00000100:00080000:3.0:1788575515.140971:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575515.140973:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575515.140975:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575515.140984:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864323200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575515.140989:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575515.140991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575515.140992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575515.140995:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7265 00000100:00080000:3.0:1788575515.140996:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575515.140998:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575515.141001:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575515.141007:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864323328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575515.141012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575515.141014:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575515.141015:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575515.141017:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7265 00000100:00080000:3.0:1788575515.141019:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575515.141023:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575515.141025:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575515.141030:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575515.141036:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864323456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575515.141066:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575515/real 1788575515] req@ffff92fc67338e00 x1875449864322944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575515.141079:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864322944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575515.141086:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575515.141089:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575515.141092:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864322944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575515.141107:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575515.141122:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575515.141130:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575515/real 1788575515] req@ffff92fc6733a680 x1875449864323072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575515.141137:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864323072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575515.141142:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575515.141144:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575515.141146:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575515.141149:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575515.141158:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575515.141163:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575515/real 1788575515] req@ffff92fc6733b100 x1875449864323200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575515.141170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864323200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575515.141174:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575515.141176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575515.141177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575515.141181:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575515.141190:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575515.141195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575515/real 1788575515] req@ffff92fc6733bb80 x1875449864323328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575515.141202:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864323328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575515.141206:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575515.141208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575515.141209:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575515.141212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575515.141221:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575515.141226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575515/real 1788575515] req@ffff92fc67338700 x1875449864323456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575515.141232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864323456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575515.141236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575515.141237:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575515.141239:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575515.588860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575515.652797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575516.612845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575516.676859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575517.636904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575517.700796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575518.660911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575518.724867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575519.684955:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575519.748869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575520.260918:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575520.260932:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575520.260937:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575520.260950:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7270 00000100:00080000:1.0:1788575520.260963:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575520.260973:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575520.261022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864323584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575520.261054:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575520.261059:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575520.261061:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575520.261065:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7270 00000100:00080000:1.0:1788575520.261069:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575520.261074:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575520.261086:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575520.261102:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864323584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575520.261102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864323712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575520.261125:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575520.261128:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575520.261130:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575520.261134:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7270 00000100:00080000:1.0:1788575520.261137:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575520.261142:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575520.261147:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575520.261160:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864323840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575520.261170:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575520.261174:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575520.261176:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575520.261179:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7270 00000100:00080000:1.0:1788575520.261184:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575520.261185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575520.261190:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575520.261197:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575520.261210:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864323968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575520.261220:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575520.261223:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575520.261225:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575520.261229:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7270 00000100:00100000:0.0:1788575520.261234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff92fc6728a680 x1875449864323584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575520.261236:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575520.261245:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575520.261252:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575520.261255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864323584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575520.261265:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788575520.261267:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864324096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575520.261270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575520.261277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864323584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575520.261306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575520.261329:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575520.261338:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff92fc67289c00 x1875449864323712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575520.261346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864323712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575520.261351:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575520.261354:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575520.261356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575520.261359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575520.261369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575520.261375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff92fc67288380 x1875449864323840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575520.261382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864323840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575520.261391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575520.261393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575520.261395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575520.261398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575520.261409:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575520.261414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff92fc67288a80 x1875449864323968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575520.261420:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864323968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575520.261425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575520.261427:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575520.261429:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864323968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575520.261432:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575520.261441:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575520.261446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575520/real 1788575520] req@ffff92fc67288700 x1875449864324096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575520.261452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864324096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575520.261462:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575520.261464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575520.261467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575520.708902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575520.772857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575521.732945:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575521.796896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575522.756830:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575522.820876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575523.780885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575523.844861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575524.804850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575524.868862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575525.380869:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575525.380881:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575525.380886:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575525.380898:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7275 00000100:00080000:1.0:1788575525.380911:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575525.380921:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575525.380996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864324224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575525.381028:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575525.381032:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575525.381034:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575525.381039:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7275 00000100:00080000:1.0:1788575525.381042:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575525.381046:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575525.381059:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575525.381074:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864324352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575525.381086:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864324224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575525.381087:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575525.381095:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575525.381098:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575525.381102:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7275 00000100:00080000:1.0:1788575525.381105:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575525.381109:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575525.381114:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575525.381127:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864324480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575525.381137:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575525.381141:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575525.381143:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575525.381147:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7275 00000100:00080000:1.0:1788575525.381159:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575525.381164:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575525.381169:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575525.381182:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864324608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575525.381191:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575525.381195:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575525.381197:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575525.381201:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7275 00000100:00080000:1.0:1788575525.381205:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575525.381210:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575525.381215:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575525.381221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575525.381228:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864324736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575525.381256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575525.381278:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575525.381301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575525.828893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575525.892892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575526.852796:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575526.916886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575527.876892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575527.940869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788575528.452977:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788575528.453010:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575528.453039:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575528.453081:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575528.453093:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575528.453102:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575528.453113:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:0.0:1788575528.453193:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff92fc67288a80 x1875449864324736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575528.453220:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864324736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575528.453232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575528.453238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575528.453241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324736:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575528.453249:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff92fc67289500 x1875449864324352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575528.456086:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864324352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575528.456096:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575528.456099:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575528.456102:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575528.456108:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff92fc67289180 x1875449864324480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575528.456115:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864324480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575528.456119:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575528.456121:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575528.456123:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575528.456126:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff92fc67288700 x1875449864324608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575528.456132:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864324608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575528.456136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575528.456138:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575528.456139:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575528.456143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575525/real 1788575528] req@ffff92fc6728bb80 x1875449864324224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575528.456151:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864324224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575528.456155:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575528.456156:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575528.456158:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864324224:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575528.900866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575528.964829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575529.924860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575529.988895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575530.500916:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.500924:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575530.500927:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.500935:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7281 00000100:00080000:0.0:1788575530.500943:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.500949:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575530.500982:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864324864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.501006:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.501009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575530.501011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.501015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7281 00000100:00080000:0.0:1788575530.501018:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.501021:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575530.501030:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575530.501043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864324992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.501062:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.501066:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575530.501068:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.501072:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7281 00000100:00080000:0.0:1788575530.501076:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.501080:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575530.501086:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575530.501103:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864324864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575530.501104:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864325120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.501120:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575530.501123:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575530.501125:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.501130:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7281 00000100:00080000:0.0:1788575530.501142:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.501147:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575530.501153:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575530.501165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864325248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575530.501177:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575530.501183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575530.501184:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575530.501188:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575530.501193:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7281 00000100:00080000:0.0:1788575530.501197:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575530.501202:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575530.501207:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575530.501218:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864325376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575530.501241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff92fc67048000 x1875449864324864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.501261:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864324864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.501272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.501275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575530.501279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864324864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575530.501286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575530.501308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575530.501319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff92fc67049f80 x1875449864324992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.501335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864324992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.501343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.501346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575530.501349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864324992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575530.501379:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575530.501396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575530.501406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff92fc67048700 x1875449864325120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.501417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864325120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.501425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.501428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575530.501431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575530.501436:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575530.501453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575530.501464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff92fc67048380 x1875449864325248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.501474:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864325248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.501487:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.501490:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575530.501492:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575530.501497:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575530.501516:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575530.501527:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575530/real 1788575530] req@ffff92fc6704a300 x1875449864325376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575530.501536:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864325376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575530.501543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575530.501547:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575530.501550:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575530.948865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575531.012829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575531.972893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575532.036836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575532.996869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575533.060813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575534.020885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575534.084836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575535.044866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575535.108849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575535.620826:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.620833:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575535.620836:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.620844:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7286 00000100:00080000:0.0:1788575535.620852:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.620859:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575535.620886:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864325504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.620910:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.620913:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575535.620916:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.620921:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7286 00000100:00080000:0.0:1788575535.620925:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.620929:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575535.620938:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575535.620939:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864325504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575535.620952:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864325632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.620960:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.620962:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575535.620963:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.620967:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7286 00000100:00080000:0.0:1788575535.620970:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.620972:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575535.620975:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575535.620983:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864325760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.620989:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.620991:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575535.620995:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575535.620995:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575535.621001:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7286 00000100:00080000:0.0:1788575535.621004:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.621006:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575535.621010:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575535.621019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864325888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.621025:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.621027:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575535.621029:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575535.621032:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff92fc67048a80 x1875449864325504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575535.621034:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7286 00000100:00080000:0.0:1788575535.621039:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.621044:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575535.621048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575535.621052:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864325504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575535.621056:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864326016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575535.621073:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.621077:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575535.621082:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864325504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575535.621099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575535.621112:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575535.621120:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff92fc6704a300 x1875449864325632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.621142:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864325632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.621148:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.621150:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575535.621152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575535.621156:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575535.621168:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575535.621175:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff92fc67048380 x1875449864325760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.621181:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864325760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.621186:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.621187:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575535.621189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575535.621193:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575535.621202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575535.621207:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff92fc67048700 x1875449864325888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.621213:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864325888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.621217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.621219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575535.621220:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864325888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575535.621223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575535.621231:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575535.621236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff92fc67049f80 x1875449864326016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.621242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864326016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.621246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.621248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575535.621249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575536.068894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575536.132842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575537.092890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575537.156808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575538.116890:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575538.180854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575539.140914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575539.204832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575540.164907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575540.228795:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575540.740819:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575540.740841:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575540.740854:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575540.740867:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7291 00000100:00080000:3.0:1788575540.740880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575540.740889:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575540.740936:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864326144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575540.740969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575540.740974:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575540.740976:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575540.740981:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7291 00000100:00080000:3.0:1788575540.740989:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575540.740993:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575540.741004:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575540.741016:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864326272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575540.741017:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864326144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575540.741030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575540.741033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575540.741034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575540.741036:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7291 00000100:00080000:3.0:1788575540.741040:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575540.741043:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575540.741046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575540.741055:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864326400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575540.741061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575540.741063:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575540.741065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575540.741068:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7291 00000100:00080000:3.0:1788575540.741069:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575540.741071:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575540.741074:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575540.741080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864326528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575540.741085:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575540.741088:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575540.741090:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575540.741093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575540.741095:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7291 00000100:00080000:3.0:1788575540.741099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575540.741101:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575540.741103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575540.741109:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864326656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575540.741146:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff92fc6733b480 x1875449864326144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.741167:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864326144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.741178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.741183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575540.741187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864326144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575540.741222:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575540.741239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575540.741247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff92fc6733a300 x1875449864326272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.741255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864326272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.741261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.741267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575540.741269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575540.741272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575540.741283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575540.741289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff92fc67339500 x1875449864326400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.741297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864326400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.741302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.741303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575540.741305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575540.741308:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575540.741318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575540.741323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff92fc6733aa00 x1875449864326528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.741330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864326528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.741334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.741336:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575540.741337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575540.741340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575540.741349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575540.741353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575540] req@ffff92fc67338000 x1875449864326656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575540.741359:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864326656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575540.741367:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575540.741369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575540.741371:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575541.188967:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575541.252792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575542.212849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575542.276798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575543.236927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575543.300806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575544.260819:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575544.324848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575545.284869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575545.348811:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575545.860812:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575545.860819:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575545.860822:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575545.860829:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7296 00000100:00080000:3.0:1788575545.860844:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575545.860852:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575545.860881:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864326784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575545.860901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575545.860904:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575545.860906:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575545.860909:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7296 00000100:00080000:3.0:1788575545.860912:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575545.860916:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575545.860925:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575545.860938:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864326912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575545.860947:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575545.860948:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864326784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575545.860949:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575545.860951:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575545.860954:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7296 00000100:00080000:3.0:1788575545.860957:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575545.860960:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575545.860964:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575545.860972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864327040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575545.860980:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575545.860982:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575545.860984:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575545.860987:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7296 00000100:00080000:3.0:1788575545.860989:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575545.860992:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575545.860996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575545.861002:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864327168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575545.861009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575545.861011:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575545.861012:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575545.861013:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575545.861016:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7296 00000100:00080000:3.0:1788575545.861018:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575545.861021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575545.861025:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575545.861032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864327296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575545.861050:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff92fc67338380 x1875449864326784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.861066:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864326784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.861076:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.861078:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575545.861081:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864326784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575545.861098:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575545.861114:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575545.861122:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff92fc67338a80 x1875449864326912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.861129:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864326912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.861134:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.861136:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575545.861138:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864326912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575545.861141:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575545.861150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575545.861155:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff92fc6733b800 x1875449864327040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.861161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864327040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.861166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.861167:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575545.861169:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575545.861172:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575545.861180:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575545.861185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff92fc67339880 x1875449864327168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.861191:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864327168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.861195:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.861197:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575545.861199:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575545.861202:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575545.861211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575545.861215:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff92fc67339c00 x1875449864327296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.861221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864327296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.861226:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.861227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575545.861229:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575546.308840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575546.372860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575547.332909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575547.396844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575548.356977:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575548.420866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575549.380921:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575549.444796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575550.404874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575550.468862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575550.980839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.980850:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575550.980854:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.980861:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7301 00000100:00080000:2.0:1788575550.980870:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.980875:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575550.980903:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864327424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.980923:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.980927:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575550.980928:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.980931:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7301 00000100:00080000:2.0:1788575550.980934:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.980938:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575550.980946:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575550.980957:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864327552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.980965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.980968:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575550.980970:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575550.980972:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864327424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575550.980973:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7301 00000100:00080000:2.0:1788575550.980977:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.980979:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575550.980983:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575550.980993:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864327680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.980999:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.981002:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575550.981003:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.981006:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7301 00000100:00080000:2.0:1788575550.981009:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.981011:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575550.981015:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575550.981023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864327808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575550.981026:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575550.981030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.981033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575550.981034:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.981037:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7301 00000100:00080000:2.0:1788575550.981039:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.981042:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575550.981045:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575550.981053:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864327936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575550.981063:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff92fc4617d180 x1875449864327424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.981078:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864327424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.981087:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.981091:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575550.981095:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864327424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575550.981115:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575550.981131:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575550.981139:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff92fc4617c380 x1875449864327552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.981146:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864327552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.981151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.981153:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575550.981155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575550.981158:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575550.981167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575550.981172:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff92fc4617d880 x1875449864327680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.981178:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864327680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.981183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.981184:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575550.981186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575550.981189:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575550.981198:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575550.981204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff92fc4617d500 x1875449864327808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.981210:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864327808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.981215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.981216:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575550.981218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575550.981221:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575550.981229:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575550.981233:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff92fc4617e300 x1875449864327936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.981239:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864327936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.981243:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.981245:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575550.981246:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864327936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575551.428867:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575551.492827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575552.452842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575552.516805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575553.476860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575553.540850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575554.500898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575554.564837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575555.524853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575555.588837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575556.100874:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575556.100880:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575556.100884:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575556.100891:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7306 00000100:00080000:0.0:1788575556.100900:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575556.100905:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575556.100934:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864328064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575556.100958:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575556.100962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575556.100964:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575556.100968:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7306 00000100:00080000:0.0:1788575556.100971:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575556.100975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575556.100985:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575556.100999:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864328192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575556.101002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864328064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575556.101013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575556.101015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575556.101017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575556.101020:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7306 00000100:00080000:0.0:1788575556.101023:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575556.101025:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575556.101031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575556.101043:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864328320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575556.101059:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575556.101059:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575556.101066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575556.101068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575556.101073:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7306 00000100:00080000:0.0:1788575556.101078:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575556.101081:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575556.101085:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575556.101096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864328448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575556.101103:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575556.101103:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575556/real 1788575556] req@ffff92fc67048000 x1875449864328064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575556.101109:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575556.101112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575556.101118:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7306 00000100:00080000:0.0:1788575556.101130:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575556.101134:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575556.101135:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864328064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575556.101137:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575556.101149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864328576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575556.101151:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575556.101159:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575556.101164:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864328064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575556.101183:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575556.101201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575556.101210:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575556/real 1788575556] req@ffff92fc6704aa00 x1875449864328192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575556.101217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864328192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575556.101223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575556.101225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575556.101227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575556.101230:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575556.101240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575556.101246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575556/real 1788575556] req@ffff92fc6704a680 x1875449864328320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575556.101253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864328320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575556.101261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575556.101263:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575556.101265:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575556.101268:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575556.101277:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575556.101282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575556/real 1788575556] req@ffff92fc6704ad80 x1875449864328448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575556.101288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864328448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575556.101292:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575556.101294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575556.101297:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575556.101302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575556.101318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575556.101328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575556/real 1788575556] req@ffff92fc67049880 x1875449864328576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575556.101339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864328576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575556.101345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575556.101347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575556.101349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575556.548866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575556.612837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575557.572856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575557.636849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575558.596873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575558.660857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575559.620840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575559.684845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575560.644932:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575560.708865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575561.220890:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575561.220902:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575561.220908:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575561.220918:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7311 00000100:00080000:3.0:1788575561.220929:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575561.220937:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575561.220979:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864328704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575561.221005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575561.221009:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575561.221011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575561.221015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7311 00000100:00080000:3.0:1788575561.221018:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575561.221022:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575561.221033:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575561.221046:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864328832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575561.221050:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864328704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575561.221066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575561.221070:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575561.221072:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575561.221076:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7311 00000100:00080000:3.0:1788575561.221080:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575561.221084:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575561.221090:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575561.221102:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864328960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575561.221118:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575561.221124:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575561.221125:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575561.221129:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575561.221133:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7311 00000100:00080000:3.0:1788575561.221137:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575561.221141:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575561.221147:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575561.221160:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864329088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575561.221169:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575561.221173:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575561.221177:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575561.221178:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff92fc67339f80 x1875449864328704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575561.221186:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7311 00000100:00080000:3.0:1788575561.221192:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575561.221197:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575561.221202:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575561.221207:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864328704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575561.221216:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864329216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575561.221224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.221232:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575561.221237:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864328704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575561.221266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575561.221283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575561.221292:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff92fc6733ad80 x1875449864328832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.221300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864328832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.221306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.221308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575561.221311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575561.221314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575561.221325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575561.221330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff92fc67339180 x1875449864328960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.221337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864328960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.221341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.221343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575561.221345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864328960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575561.221348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575561.221358:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575561.221363:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff92fc67339c00 x1875449864329088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.221375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864329088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.221380:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.221382:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575561.221383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575561.221386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575561.221396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575561.221402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575561/real 1788575561] req@ffff92fc67339880 x1875449864329216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575561.221409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864329216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575561.221413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575561.221415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575561.221417:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575561.668870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575561.732932:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575562.692925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575562.756826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575563.716871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575563.780822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575564.740850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575564.804819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575565.764859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575565.828825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575566.340861:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.340871:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575566.340875:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.340885:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7316 00000100:00080000:0.0:1788575566.340895:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.340903:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575566.340944:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864329344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.340966:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.340969:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575566.340970:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.340973:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7316 00000100:00080000:0.0:1788575566.340975:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.340977:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575566.340984:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575566.340990:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864329472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.340997:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.341000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575566.341001:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.341004:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7316 00000100:00080000:0.0:1788575566.341008:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.341011:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575566.341015:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575566.341022:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864329344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575566.341027:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864329600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.341038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.341042:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575566.341043:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.341047:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7316 00000100:00080000:0.0:1788575566.341051:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575566.341054:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575566.341058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575566.341067:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864329728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575566.341073:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575566.341075:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575566.341076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575566.341078:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7316 00000100:00080000:0.0:1788575566.341083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575566.341083:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575566.341089:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575566.341092:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575566.341101:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864329856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575566.341127:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff92fc67049c00 x1875449864329344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.341148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864329344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.341158:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.341161:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575566.341165:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864329344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575566.341171:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575566.341189:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575566.341199:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff92fc67048e00 x1875449864329472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.341209:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864329472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.341217:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.341219:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575566.341221:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575566.341238:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575566.341251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575566.341257:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff92fc67049180 x1875449864329600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.341264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864329600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.341268:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.341270:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575566.341272:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575566.341275:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575566.341283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575566.341288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff92fc6704bb80 x1875449864329728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.341300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864329728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.341305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.341306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575566.341308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575566.341311:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575566.341320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575566.341325:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575566/real 1788575566] req@ffff92fc67049500 x1875449864329856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575566.341335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864329856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575566.341342:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575566.341345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575566.341348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864329856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575566.788885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575566.852846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575567.812871:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575567.876901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575568.836919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575568.900873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575569.860932:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575569.924870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575570.884893:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575570.948883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575571.460856:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575571.460871:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575571.460876:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575571.460887:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7322 00000100:00080000:0.0:1788575571.460900:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575571.460908:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575571.460954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864329984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575571.460983:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575571.460987:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575571.460992:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575571.460999:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7322 00000100:00080000:0.0:1788575571.461003:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575571.461007:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575571.461019:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575571.461021:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864329984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575571.461035:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864330112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575571.461045:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575571.461048:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575571.461051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575571.461056:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7322 00000100:00080000:0.0:1788575571.461060:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575571.461065:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575571.461073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575571.461087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864330240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575571.461095:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575571.461098:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575571.461100:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575571.461102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575571.461105:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7322 00000100:00080000:0.0:1788575571.461108:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575571.461122:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575571.461131:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575571.461141:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864330368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575571.461149:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575571.461151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff92fc6704b100 x1875449864329984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575571.461152:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575571.461154:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575571.461157:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7322 00000100:00080000:0.0:1788575571.461161:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575571.461163:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575571.461167:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575571.461313:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864330496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575571.461367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864329984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575571.461377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575571.461381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575571.461385:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864329984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575571.461455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575571.461477:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575571.461487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff92fc67049500 x1875449864330112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575571.461495:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864330112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575571.461501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575571.461503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575571.461505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575571.461508:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575571.461519:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575571.461525:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff92fc6704bb80 x1875449864330240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575571.461532:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864330240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575571.461541:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575571.461543:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575571.461545:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575571.461548:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575571.461558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575571.461563:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff92fc67049180 x1875449864330368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575571.461569:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864330368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575571.461574:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575571.461575:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575571.461577:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575571.461581:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575571.461590:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575571.461596:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575571/real 1788575571] req@ffff92fc67048e00 x1875449864330496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575571.461602:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864330496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575571.461607:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575571.461608:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575571.461611:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575571.908843:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575571.972838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575572.932897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575572.996941:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575573.956845:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575574.020822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575574.980873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575575.044851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575576.004903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575576.068808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575576.580860:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575576.580868:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575576.580871:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575576.580881:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7327 00000100:00080000:3.0:1788575576.580891:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575576.580898:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575576.580954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864330624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575576.580984:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575576.580988:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575576.580990:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575576.580993:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7327 00000100:00080000:3.0:1788575576.580997:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575576.581000:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575576.581009:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575576.581021:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864330752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575576.581031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575576.581034:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575576.581038:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864330624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575576.581038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575576.581046:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7327 00000100:00080000:3.0:1788575576.581052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575576.581057:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575576.581062:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575576.581073:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864330880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575576.581087:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575576.581090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575576.581092:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575576.581095:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7327 00000100:00080000:3.0:1788575576.581099:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575576.581102:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575576.581107:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575576.581110:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575576.581120:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864331008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575576.581130:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575576.581133:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575576.581134:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575576.581138:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7327 00000100:00080000:3.0:1788575576.581141:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575576.581145:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575576.581150:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575576.581151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff92fc6733b800 x1875449864330624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575576.581163:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864331136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575576.581172:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864330624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.581183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.581188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575576.581193:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864330624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575576.581214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575576.581232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575576.581242:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff92fc67338a80 x1875449864330752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.581253:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864330752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.581261:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.581264:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575576.581267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575576.581272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575576.581285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575576.581291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff92fc67338380 x1875449864330880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.581302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864330880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.581306:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.581308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575576.581310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864330880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575576.581312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575576.581322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575576.581327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff92fc67338000 x1875449864331008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.581333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864331008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.581337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.581339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575576.581340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575576.581343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575576.581352:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575576.581361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff92fc6733aa00 x1875449864331136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575576.581371:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864331136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575576.581383:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575576.581386:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575576.581389:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575577.028918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575577.092863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575578.052910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575578.116863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575579.076886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575579.140838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575580.100995:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575580.164888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575581.124878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575581.188876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575581.700861:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575581.700868:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575581.700871:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575581.700880:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7332 00000100:00080000:0.0:1788575581.700890:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575581.700896:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575581.700929:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864331264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575581.700948:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575581.700965:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575581.700967:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575581.700970:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7332 00000100:00080000:0.0:1788575581.700973:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575581.700976:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575581.700984:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575581.700989:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864331264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575581.700996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864331392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575581.701008:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575581.701010:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575581.701011:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575581.701014:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7332 00000100:00080000:0.0:1788575581.701016:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575581.701018:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575581.701021:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575581.701028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864331520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575581.701033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575581.701036:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575581.701037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575581.701040:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575581.701043:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7332 00000100:00080000:0.0:1788575581.701047:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575581.701049:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575581.701053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575581.701060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864331648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575581.701066:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575581.701068:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575581.701070:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575581.701073:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7332 00000100:00080000:0.0:1788575581.701076:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575581.701078:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575581.701081:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575581.701086:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff92fc67049c00 x1875449864331264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575581.701089:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864331776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575581.701116:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864331264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.701125:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.701128:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575581.701131:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864331264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575581.701147:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575581.701160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575581.701168:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff92fc67049880 x1875449864331392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.701175:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864331392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.701181:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.701183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575581.701185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575581.701188:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575581.701196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575581.701202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff92fc6704ad80 x1875449864331520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.701208:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864331520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.701212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.701214:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575581.701216:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575581.701218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575581.701227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575581.701232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff92fc6704a680 x1875449864331648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.701238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864331648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.701242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.701244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575581.701246:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575581.701249:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575581.701258:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575581.701263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff92fc6704aa00 x1875449864331776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.701269:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864331776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.701273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.701274:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575581.701276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864331776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788575582.148918:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575582.212828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575583.172850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575583.236858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575584.196884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575584.260822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575585.220866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575585.284771:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575586.244903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575586.308843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575586.820813:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.820821:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575586.820825:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.820833:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7337 00000100:00080000:2.0:1788575586.820842:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.820855:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575586.820891:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864331904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575586.820914:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.820917:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575586.820919:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.820922:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7337 00000100:00080000:2.0:1788575586.820926:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.820943:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575586.820952:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575586.820964:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864331904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575586.820966:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864332032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575586.820988:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.820992:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575586.820994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.820998:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7337 00000100:00080000:2.0:1788575586.821002:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.821007:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575586.821012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575586.821025:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864332160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575586.821033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575586.821039:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575586.821043:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575586.821045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.821050:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7337 00000100:00080000:2.0:1788575586.821054:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575586.821058:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575586.821063:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575586.821075:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864332288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575586.821083:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575586.821087:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff92fc4617ea00 x1875449864331904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575586.821087:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575586.821091:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575586.821099:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7337 00000100:00080000:2.0:1788575586.821104:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575586.821109:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864331904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575586.821110:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575586.821118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575586.821128:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.821133:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:2.0:1788575586.821133:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864332416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575586.821140:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864331904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575586.821165:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575586.821182:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575586.821190:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff92fc4617ce00 x1875449864332032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.821197:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864332032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.821203:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.821205:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575586.821207:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575586.821210:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575586.821219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575586.821227:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff92fc4617e300 x1875449864332160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.821237:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864332160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.821244:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.821247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575586.821250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575586.821256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575586.821274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575586.821282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff92fc4617d500 x1875449864332288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.821297:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864332288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.821302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.821306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575586.821308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575586.821312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575586.821323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575586.821329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff92fc4617d880 x1875449864332416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.821336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864332416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.821340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.821342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575586.821344:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575587.268858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575587.332835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575588.292892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575588.356818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575589.316837:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575589.380843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575590.340982:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575590.404830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575591.364892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575591.428821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575591.940903:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575591.940916:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575591.940923:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575591.940936:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7342 00000100:00080000:2.0:1788575591.940971:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575591.940982:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575591.941031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864332544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575591.941065:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575591.941068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575591.941070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575591.941074:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7342 00000100:00080000:2.0:1788575591.941078:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575591.941082:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575591.941095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575591.941109:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864332672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575591.941119:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575591.941123:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575591.941125:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575591.941128:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7342 00000100:00080000:2.0:1788575591.941132:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575591.941136:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575591.941140:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575591.941151:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864332544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575591.941155:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864332800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575591.941179:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575591.941182:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575591.941184:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575591.941187:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7342 00000100:00080000:2.0:1788575591.941191:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575591.941196:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575591.941202:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575591.941216:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864332928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575591.941226:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575591.941230:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575591.941232:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575591.941237:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7342 00000100:00080000:2.0:1788575591.941241:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575591.941245:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575591.941250:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575591.941264:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864333056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575591.941286:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575591.941332:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575591.941360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575591.941384:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788575592.388912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575592.452824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575593.412854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575593.476782:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575594.437006:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575594.500900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788575595.013139:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788575595.013230:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575595.013254:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575595.013325:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575595.013337:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575595.013346:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575595.013355:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788575595.013479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575595] req@ffff92fc4617c000 x1875449864332800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575595.013514:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864332800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575595.013532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575595.013539:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575595.013544:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575595.013554:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575595] req@ffff92fc4617fb80 x1875449864332928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575595.013567:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864332928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575595.013575:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575595.013579:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575595.013583:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575595.013592:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575595] req@ffff92fc4617e680 x1875449864333056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575595.013612:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864333056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575595.013619:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575595.013623:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575595.013626:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333056:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575595.013645:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575595] req@ffff92fc4617c380 x1875449864332544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575595.013659:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864332544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575595.013668:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575595.013672:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575595.013676:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864332544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575595.013683:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575595] req@ffff92fc4617d180 x1875449864332672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575595.013702:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864332672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575595.013712:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575595.013716:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575595.013887:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864332672:192.168.202.132@tcp:38:kworker.0 00010000:00010000:0.0:1788575595.460905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575595.524831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575596.484909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575596.548832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575597.060836:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575597.060845:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575597.060849:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575597.060858:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7347 00000100:00080000:3.0:1788575597.060868:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575597.060874:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575597.060907:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864333184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575597.060940:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575597.060944:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575597.060946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575597.060950:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7347 00000100:00080000:3.0:1788575597.060954:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575597.060957:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575597.060976:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575597.060991:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864333312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575597.061001:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575597.061010:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575597.061012:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575597.061015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7347 00000100:00080000:3.0:1788575597.061022:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575597.061026:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575597.061031:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575597.061042:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864333440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575597.061050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575597.061053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575597.061054:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575597.061057:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7347 00000100:00080000:3.0:1788575597.061061:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575597.061064:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575597.061069:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575597.061078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864333568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575597.061087:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575597.061090:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575597.061091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575597.061094:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7347 00000100:00080000:3.0:1788575597.061098:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575597.061101:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575597.061105:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575597.061115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864333696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575597.061207:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864333184:192.168.202.132@tcp:250:kworker.0 00000800:00000100:1.0:1788575597.061275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575597.061317:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575597/real 1788575597] req@ffff92fc67339500 x1875449864333184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575597.061335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864333184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575597.061345:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575597.061349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575597.061353:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864333184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575597.061359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575597.061380:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575597.061392:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575597/real 1788575597] req@ffff92fc6733a300 x1875449864333312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575597.061408:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864333312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575597.061415:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575597.061417:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575597.061420:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575597.061425:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575597.061444:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575597.061454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575597/real 1788575597] req@ffff92fc6733b480 x1875449864333440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575597.061465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864333440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575597.061472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575597.061475:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575597.061478:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575597.061484:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575597.061503:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575597.061513:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575597/real 1788575597] req@ffff92fc67338700 x1875449864333568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575597.061523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864333568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575597.061536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575597.061539:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575597.061542:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575597.061547:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575597.061566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575597.061576:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575597/real 1788575597] req@ffff92fc6733bb80 x1875449864333696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575597.061586:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864333696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575597.061592:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575597.061595:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575597.061598:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575597.508855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575597.572837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575598.532881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575598.596816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575599.556840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575599.620840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575600.580842:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575600.644836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575601.604863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575601.668830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575602.180841:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575602.180848:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575602.180852:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575602.180861:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7352 00000100:00080000:0.0:1788575602.180870:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575602.180878:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575602.180912:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864333824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575602.180935:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575602.180939:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575602.180940:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575602.180945:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7352 00000100:00080000:0.0:1788575602.180948:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575602.180952:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575602.180961:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575602.180972:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864333952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575602.180978:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864333824:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575602.180993:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575602.180996:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575602.180998:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575602.181001:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7352 00000100:00080000:0.0:1788575602.181004:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575602.181007:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575602.181011:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575602.181019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864334080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575602.181031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575602.181033:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575602.181035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575602.181038:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7352 00000100:00080000:0.0:1788575602.181040:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575602.181041:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575602.181043:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575602.181047:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575602.181054:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864334208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575602.181061:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575602.181064:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575602.181065:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575602.181068:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7352 00000100:00080000:0.0:1788575602.181070:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575602.181073:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575602.181076:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff92fc67048000 x1875449864333824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575602.181077:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575602.181083:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864334336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575602.181091:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864333824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575602.181100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575602.181103:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575602.181107:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864333824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575602.181128:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575602.181144:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575602.181152:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff92fc67049f80 x1875449864333952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575602.181160:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864333952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575602.181168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575602.181171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575602.181174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864333952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575602.181179:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575602.181196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575602.181212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff92fc67048700 x1875449864334080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575602.181223:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864334080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575602.181231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575602.181234:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575602.181236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575602.181241:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575602.181261:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575602.181271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff92fc67048380 x1875449864334208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575602.181283:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864334208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575602.181290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575602.181293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575602.181296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575602.181301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575602.181317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575602.181323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575602/real 1788575602] req@ffff92fc6704a300 x1875449864334336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575602.181330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864334336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575602.181334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575602.181337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575602.181340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788575602.628863:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575602.692848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575603.652878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575603.716829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575604.676905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575604.740837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575605.700905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575605.764809:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575606.725006:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575606.788857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575607.300901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575607.300914:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575607.300919:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575607.300932:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7357 00000100:00080000:0.0:1788575607.300944:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575607.300952:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575607.301010:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864334464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575607.301036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575607.301039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575607.301041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575607.301044:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7357 00000100:00080000:0.0:1788575607.301046:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575607.301048:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575607.301058:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575607.301070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864334592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575607.301077:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575607.301080:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575607.301081:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575607.301084:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7357 00000100:00080000:0.0:1788575607.301086:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575607.301090:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575607.301091:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864334464:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788575607.301095:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575607.301105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864334720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575607.301111:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575607.301113:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575607.301114:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575607.301116:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7357 00000100:00080000:0.0:1788575607.301118:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575607.301120:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575607.301123:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575607.301129:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864334848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575607.301134:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575607.301136:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575607.301137:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575607.301139:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7357 00000100:00080000:0.0:1788575607.301160:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575607.301162:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575607.301165:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575607.301173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575607.301174:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864334976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575607.301241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff92fc67048a80 x1875449864334464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575607.301264:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864334464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575607.301282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575607.301287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575607.301291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864334464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575607.301319:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575607.301349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575607.301364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff92fc6704a300 x1875449864334592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575607.301380:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864334592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575607.301390:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575607.301394:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575607.301397:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575607.301405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575607.301426:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575607.301438:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff92fc67048380 x1875449864334720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575607.301451:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864334720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575607.301460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575607.301464:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575607.301469:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575607.301476:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575607.301500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575607.301511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff92fc67048700 x1875449864334848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575607.301519:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864334848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575607.301525:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575607.301528:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575607.301530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575607.301534:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575607.301553:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575607.301561:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575607/real 1788575607] req@ffff92fc67049f80 x1875449864334976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575607.301568:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864334976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575607.301583:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575607.301587:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575607.301590:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864334976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575607.748946:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575607.812878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575608.772897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575608.836910:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575609.796855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575609.860874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575610.820960:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575610.884849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575611.844897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575611.908864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575612.420840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575612.420858:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575612.420863:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575612.420876:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7363 00000100:00080000:2.0:1788575612.420888:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575612.420897:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575612.420938:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864335104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575612.420966:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575612.420969:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575612.420971:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575612.420975:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7363 00000100:00080000:2.0:1788575612.420977:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575612.420981:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575612.420993:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575612.421022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864335232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575612.421032:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864335104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575612.421036:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575612.421041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575612.421043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575612.421047:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7363 00000100:00080000:2.0:1788575612.421051:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575612.421055:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575612.421060:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575612.421072:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864335360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575612.421081:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575612.421085:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575612.421087:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575612.421090:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7363 00000100:00080000:2.0:1788575612.421094:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575612.421097:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575612.421102:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575612.421111:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864335488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575612.421119:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575612.421124:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575612.421128:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575612.421130:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575612.421134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7363 00000100:00080000:2.0:1788575612.421137:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575612.421141:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575612.421145:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575612.421154:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864335616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575612.421178:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff92fc4617f800 x1875449864335104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575612.421200:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864335104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575612.421211:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575612.421216:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575612.421220:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864335104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575612.421248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575612.421272:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575612.421287:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff92fc4617f100 x1875449864335232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575612.421300:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864335232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575612.421309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575612.421312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575612.421316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575612.421322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575612.421342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575612.421353:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff92fc4617dc00 x1875449864335360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575612.421365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864335360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575612.421379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575612.421383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575612.421386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575612.421392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575612.421413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575612.421424:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff92fc4617c700 x1875449864335488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575612.421436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864335488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575612.421444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575612.421447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575612.421450:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575612.421456:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575612.421476:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575612.421487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575612/real 1788575612] req@ffff92fc4617ca80 x1875449864335616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575612.421498:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864335616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575612.421506:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575612.421509:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575612.421513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:3.0:1788575612.868950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575612.932871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575613.892885:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575613.956833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575614.917055:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575614.981160:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575615.940925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575616.004889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:3.0:1788575616.964948:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575617.028889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575617.540870:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575617.540886:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575617.540891:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575617.540907:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7368 00000100:00080000:2.0:1788575617.540921:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575617.540932:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575617.540980:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864335744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575617.541011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575617.541015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575617.541017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575617.541021:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7368 00000100:00080000:2.0:1788575617.541036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575617.541041:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575617.541053:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575617.541066:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864335872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575617.541075:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864335744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575617.541076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575617.541079:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575617.541081:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575617.541084:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7368 00000100:00080000:2.0:1788575617.541087:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575617.541091:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575617.541095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575617.541106:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864336000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575617.541114:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575617.541117:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575617.541118:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575617.541121:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7368 00000100:00080000:2.0:1788575617.541124:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575617.541127:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575617.541131:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575617.541140:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864336128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575617.541147:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575617.541150:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575617.541151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575617.541152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575617.541155:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7368 00000100:00080000:2.0:1788575617.541157:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575617.541160:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575617.541163:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575617.541171:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864336256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575617.541202:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff92fc4617ed80 x1875449864335744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575617.541221:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864335744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575617.541232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575617.541236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575617.541240:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864335744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575617.541264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575617.541282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575617.541291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff92fc4617f480 x1875449864335872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575617.541298:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864335872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575617.541303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575617.541305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575617.541307:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864335872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575617.541310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575617.541321:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575617.541327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff92fc4617ca80 x1875449864336000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575617.541333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864336000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575617.541338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575617.541339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575617.541342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575617.541347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575617.541364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575617.541374:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff92fc4617c700 x1875449864336128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575617.541396:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864336128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575617.541404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575617.541407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575617.541410:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575617.541415:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575617.541436:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575617.541447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575617/real 1788575617] req@ffff92fc4617dc00 x1875449864336256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575617.541458:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864336256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575617.541466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575617.541469:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575617.541472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575617.988947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575618.052887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575619.012965:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575619.076861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575620.036851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575620.100891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575621.060978:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575621.124939:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575622.084887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575622.148863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575622.660927:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.660941:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575622.660946:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.660960:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7373 00000100:00080000:0.0:1788575622.660974:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.660983:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575622.661033:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864336384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575622.661087:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.661091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575622.661093:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.661097:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7373 00000100:00080000:0.0:1788575622.661101:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.661104:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575622.661116:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575622.661130:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864336512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575622.661139:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864336384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575622.661143:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.661149:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575622.661151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.661155:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7373 00000100:00080000:0.0:1788575622.661158:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.661162:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575622.661167:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575622.661179:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864336640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575622.661188:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.661191:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575622.661193:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.661197:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7373 00000100:00080000:0.0:1788575622.661211:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.661216:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575622.661221:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575622.661227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575622.661237:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864336768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575622.661248:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575622.661251:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575622.661253:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575622.661257:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7373 00000100:00080000:0.0:1788575622.661261:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575622.661265:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575622.661271:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575622.661279:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff92fc67048000 x1875449864336384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575622.661285:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864336896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575622.661302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864336384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.661313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.661316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575622.661319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864336384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575622.661353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575622.661375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575622.661388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff92fc6704aa00 x1875449864336512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.661400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864336512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.661406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.661409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575622.661411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575622.661416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575622.661436:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575622.661447:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff92fc6704a680 x1875449864336640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.661459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864336640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.661466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.661469:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575622.661471:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575622.661475:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575622.661489:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575622.661495:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff92fc6704ad80 x1875449864336768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.661502:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864336768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.661507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.661509:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575622.661512:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575622.661518:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575622.661534:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575622.661544:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff92fc67049880 x1875449864336896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575622.661555:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864336896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575622.661563:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575622.661566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575622.661569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864336896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575623.108915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575623.172827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575624.132879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575624.196836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575625.156872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575625.220869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575626.180914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575626.244867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575627.204873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575627.268843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575627.780895:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.780905:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575627.780911:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.780921:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7378 00000100:00080000:0.0:1788575627.780931:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.780939:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575627.780976:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864337024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.781021:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.781025:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575627.781027:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.781031:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7378 00000100:00080000:0.0:1788575627.781035:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.781040:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575627.781050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575627.781078:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864337152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575627.781078:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864337024:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575627.781088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.781094:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575627.781096:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.781099:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7378 00000100:00080000:0.0:1788575627.781102:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.781105:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575627.781110:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575627.781121:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864337280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.781134:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.781138:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788575627.781150:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575627.781151:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.781159:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7378 00000100:00080000:0.0:1788575627.781164:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.781169:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575627.781174:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575627.781186:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864337408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575627.781192:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff92fc67049c00 x1875449864337024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575627.781218:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575627.781222:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864337024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575627.781224:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575627.781229:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.781239:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7378 00000100:00080000:1.0:1788575627.781242:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575627.781244:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575627.781248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:0.0:1788575627.781253:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575627.781256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864337024:192.168.202.132@tcp:250:kworker.0 00000008:00100000:0.0:1788575627.781261:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575627.781275:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864337536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575627.781283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575627.781308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575627.781321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff92fc67048e00 x1875449864337152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.781331:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864337152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.781343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.781345:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575627.781348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575627.781353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575627.781368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575627.781377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff92fc67049180 x1875449864337280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.781387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864337280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.781394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.781396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575627.781399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575627.781404:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575627.781419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575627.781428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff92fc6704bb80 x1875449864337408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.781437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864337408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.781449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.781452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575627.781455:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575627.781467:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575627.781483:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575627.781492:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff92fc67049500 x1875449864337536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575627.781501:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864337536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575627.781508:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575627.781510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575627.781513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575628.228926:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575628.292849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575629.252920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575629.316855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575630.276931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575630.340807:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575631.300874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575631.364860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575632.324873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575632.388855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575632.900839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.900849:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575632.900853:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.900862:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7383 00000100:00080000:0.0:1788575632.900871:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.900878:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575632.900912:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864337664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.900944:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.900948:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575632.900949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.900953:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7383 00000100:00080000:0.0:1788575632.900956:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.900960:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575632.900968:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575632.900981:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864337792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.900990:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.900995:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575632.900995:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864337664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575632.900999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.901006:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7383 00000100:00080000:0.0:1788575632.901010:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.901013:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575632.901018:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575632.901029:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864337920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.901038:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.901041:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575632.901043:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.901046:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7383 00000100:00080000:0.0:1788575632.901049:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.901053:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575632.901057:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575632.901059:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575632.901095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864338048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.901107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.901110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575632.901114:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575632.901114:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff92fc6704b100 x1875449864337664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575632.901121:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7383 00000100:00080000:0.0:1788575632.901127:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.901133:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575632.901138:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575632.901139:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864337664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575632.901154:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864338176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575632.901154:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.901162:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575632.901169:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864337664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575632.901193:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575632.901214:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575632.901226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff92fc67049500 x1875449864337792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.901237:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864337792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.901246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.901248:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575632.901251:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575632.901256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575632.901271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575632.901281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff92fc6704bb80 x1875449864337920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.901302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864337920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.901310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.901313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575632.901316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864337920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575632.901322:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575632.901338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575632.901347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff92fc67049180 x1875449864338048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.901358:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864338048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.901366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.901369:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575632.901372:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575632.901377:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575632.901392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575632.901402:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff92fc67048e00 x1875449864338176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575632.901412:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864338176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575632.901419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575632.901422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575632.901425:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575633.349225:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575633.412856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575634.372865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575634.436844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575635.396878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575635.460835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575636.420871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575636.484849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575637.444973:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575637.508893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575638.020923:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575638.020940:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575638.020946:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575638.020962:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7388 00000100:00080000:3.0:1788575638.020978:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575638.020989:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575638.021047:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864338304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575638.021080:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575638.021097:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575638.021100:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575638.021104:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7388 00000100:00080000:3.0:1788575638.021108:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575638.021113:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575638.021125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575638.021140:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864338432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575638.021152:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575638.021157:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575638.021160:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575638.021162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864338304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575638.021165:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7388 00000100:00080000:3.0:1788575638.021172:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575638.021178:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575638.021183:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575638.021197:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864338560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575638.021207:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575638.021210:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575638.021212:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575638.021217:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7388 00000100:00080000:3.0:1788575638.021220:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575638.021225:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575638.021231:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575638.021245:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864338688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575638.021255:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575638.021257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575638.021262:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575638.021266:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575638.021270:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7388 00000100:00080000:3.0:1788575638.021275:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575638.021279:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575638.021285:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575638.021298:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864338816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575638.021309:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575638/real 1788575638] req@ffff92fc6733b100 x1875449864338304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575638.021327:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864338304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575638.021335:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575638.021339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575638.021343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864338304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575638.021368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575638.021383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575638.021394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575638/real 1788575638] req@ffff92fc6733a680 x1875449864338432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575638.021406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864338432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575638.021414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575638.021418:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575638.021421:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575638.021427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575638.021447:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575638.021455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575638/real 1788575638] req@ffff92fc67338e00 x1875449864338560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575638.021462:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864338560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575638.021468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575638.021470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575638.021472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575638.021475:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575638.021487:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575638.021493:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575638/real 1788575638] req@ffff92fc6733bb80 x1875449864338688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575638.021500:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864338688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575638.021505:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575638.021506:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575638.021508:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575638.021511:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575638.021521:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575638.021526:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575638/real 1788575638] req@ffff92fc67338700 x1875449864338816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575638.021533:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864338816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575638.021537:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575638.021538:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575638.021540:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864338816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575638.468931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575638.532892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575639.493050:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575639.556848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575640.516855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575640.580839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575641.540881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575641.604837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575642.564869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575642.628844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575643.140827:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575643.140837:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575643.140841:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575643.140849:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7393 00000100:00080000:3.0:1788575643.140858:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575643.140864:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575643.140900:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864338944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575643.140924:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575643.140928:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575643.140929:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575643.140933:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7393 00000100:00080000:3.0:1788575643.140937:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575643.140941:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575643.140950:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575643.140961:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864339072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575643.140970:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575643.140973:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575643.140974:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575643.140977:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7393 00000100:00100000:1.0:1788575643.140980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864338944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575643.140980:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575643.140984:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575643.140987:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575643.140996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864339200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575643.141003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575643.141005:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575643.141007:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575643.141009:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7393 00000100:00080000:3.0:1788575643.141012:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575643.141015:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575643.141018:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575643.141026:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864339328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575643.141033:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575643.141033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575643.141036:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575643.141037:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575643.141040:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7393 00000100:00080000:3.0:1788575643.141042:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575643.141045:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575643.141048:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575643.141055:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864339456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575643.141070:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575643/real 1788575643] req@ffff92fc6733b480 x1875449864338944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575643.141085:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864338944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575643.141100:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575643.141117:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575643.141120:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864338944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575643.141127:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575643.141148:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575643.141160:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575643/real 1788575643] req@ffff92fc6733a300 x1875449864339072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575643.141170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864339072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575643.141178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575643.141180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575643.141183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575643.141205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575643.141224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575643.141235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575643/real 1788575643] req@ffff92fc67339500 x1875449864339200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575643.141246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864339200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575643.141253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575643.141255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575643.141258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575643.141263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575643.141285:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575643.141296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575643/real 1788575643] req@ffff92fc6733aa00 x1875449864339328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575643.141308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864339328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575643.141316:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575643.141319:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575643.141323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575643.141329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575643.141349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575643.141361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575643/real 1788575643] req@ffff92fc67338000 x1875449864339456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575643.141374:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864339456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575643.141381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575643.141384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575643.141387:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575643.588846:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575643.652822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575644.612870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575644.676840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575645.636854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575645.700822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575646.660856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575646.724855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575647.684852:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575647.748806:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575648.260821:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.260831:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575648.260835:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.260845:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7398 00000100:00080000:3.0:1788575648.260854:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.260862:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575648.260903:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864339584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.260926:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.260928:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575648.260929:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.260932:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7398 00000100:00080000:3.0:1788575648.260934:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.260937:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575648.260944:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575648.260954:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864339712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.260962:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.260964:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575648.260965:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.260969:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7398 00000100:00100000:1.0:1788575648.260971:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864339584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575648.260971:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.260975:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575648.260982:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575648.260994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864339840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.261003:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.261007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575648.261008:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.261013:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7398 00000100:00080000:3.0:1788575648.261017:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.261021:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575648.261027:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575648.261030:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575648.261041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864339968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575648.261050:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575648.261053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575648.261055:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575648.261058:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7398 00000100:00080000:3.0:1788575648.261062:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575648.261065:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575648.261070:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575648.261071:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff92fc67338380 x1875449864339584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575648.261081:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864340096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575648.261089:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864339584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.261101:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.261105:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575648.261110:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864339584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575648.261167:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575648.261185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575648.261197:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff92fc67338a80 x1875449864339712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.261216:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864339712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.261225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.261228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575648.261231:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575648.261237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575648.261257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575648.261265:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff92fc6733b800 x1875449864339840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.261274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864339840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.261287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.261290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575648.261293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575648.261298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575648.261317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575648.261326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff92fc67339880 x1875449864339968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.261332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864339968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.261337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.261339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575648.261340:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864339968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575648.261344:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575648.261354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575648.261359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575648/real 1788575648] req@ffff92fc67339c00 x1875449864340096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575648.261365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864340096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575648.261369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575648.261371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575648.261373:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575648.708853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575648.772864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575649.732885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575649.796804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575650.756883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575650.820832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575651.780850:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575651.844801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575652.804858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575652.868820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575653.380865:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575653.380874:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575653.380878:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575653.380888:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7403 00000100:00080000:0.0:1788575653.380897:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575653.380904:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575653.380939:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864340224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575653.380960:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575653.380967:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575653.380969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575653.380972:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7403 00000100:00080000:0.0:1788575653.380976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575653.380980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575653.380989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575653.381001:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864340352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575653.381010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864340224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575653.381026:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575653.381028:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575653.381029:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575653.381033:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7403 00000100:00080000:0.0:1788575653.381036:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575653.381039:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575653.381043:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575653.381052:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864340480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575653.381057:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575653.381060:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575653.381061:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575653.381063:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7403 00000100:00080000:0.0:1788575653.381065:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575653.381070:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575653.381071:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788575653.381073:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575653.381082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864340608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575653.381089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575653.381091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575653.381092:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575653.381093:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7403 00000100:00080000:0.0:1788575653.381096:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575653.381098:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575653.381101:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575653.381110:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff92fc67049c00 x1875449864340224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575653.381111:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864340736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575653.381165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864340224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.381176:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.381180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575653.381183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864340224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575653.381205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575653.381227:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575653.381239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff92fc67049880 x1875449864340352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575653.381251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864340352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.381260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.381263:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575653.381267:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575653.381272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575653.381288:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575653.381298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff92fc6704ad80 x1875449864340480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575653.381323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864340480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.381330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.381334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575653.381337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575653.381343:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575653.381361:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575653.381371:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff92fc6704a680 x1875449864340608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575653.381381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864340608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.381389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.381392:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575653.381395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575653.381400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575653.381426:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575653.381436:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575653/real 1788575653] req@ffff92fc6704aa00 x1875449864340736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575653.381448:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864340736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575653.381455:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575653.381458:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575653.381462:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575653.828861:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575653.892827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575654.852858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575654.916876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575655.876870:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575655.940840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575656.901000:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575656.964868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575657.924889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575657.988860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575658.500863:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.500875:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575658.500881:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.500894:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7409 00000100:00080000:0.0:1788575658.500907:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.500915:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575658.500975:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864340864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575658.501006:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.501010:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575658.501012:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.501017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7409 00000100:00080000:0.0:1788575658.501021:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.501026:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575658.501042:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575658.501057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864340992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575658.501067:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864340864:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575658.501073:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.501082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575658.501085:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.501089:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7409 00000100:00080000:0.0:1788575658.501093:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.501096:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575658.501101:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575658.501118:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864341120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575658.501140:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.501143:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575658.501145:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575658.501150:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7409 00000100:00080000:0.0:1788575658.501155:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.501159:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575658.501165:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575658.501178:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864341248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575658.501189:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575658.501193:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575658.501196:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575658.501201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340992:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788575658.501204:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7409 00000100:00080000:0.0:1788575658.501215:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575658.501221:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575658.501227:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575658.501246:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864341376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575658.501263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575658.501284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575658.501301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575658.948840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575659.012880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575659.974359:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575660.036893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575660.996908:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575661.060853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788575661.573259:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788575661.573313:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575661.573328:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575661.573368:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575661.573376:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575661.573384:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575661.573394:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575661.573428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff92fc6704a300 x1875449864341376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.573454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864341376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.573464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.573468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575661.573472:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341376:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575661.573479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff92fc67049f80 x1875449864340992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.573489:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864340992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.573501:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.573504:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575661.573507:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864340992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575661.573513:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff92fc67048700 x1875449864341120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.573523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864341120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.573530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.573534:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575661.573536:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575661.573543:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff92fc67048380 x1875449864341248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.573553:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864341248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.573561:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.573564:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575661.573567:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575661.573573:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575658/real 1788575661] req@ffff92fc67048000 x1875449864340864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575661.573585:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864340864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575661.573593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575661.573596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575661.573599:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864340864:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575662.020909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575662.084848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575663.044919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575663.108830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575663.620912:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.620922:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575663.620930:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.620939:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7414 00000100:00080000:2.0:1788575663.620950:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.620956:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575663.620987:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864341504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575663.621016:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.621022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575663.621023:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.621026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7414 00000100:00080000:2.0:1788575663.621029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.621032:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575663.621041:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575663.621053:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864341632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575663.621063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.621067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575663.621070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575663.621071:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864341504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575663.621075:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7414 00000100:00080000:2.0:1788575663.621081:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.621084:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575663.621088:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575663.621101:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864341760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575663.621109:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.621112:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575663.621114:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.621118:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7414 00000100:00080000:2.0:1788575663.621138:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.621143:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575663.621149:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575663.621153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575663.621163:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864341888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575663.621173:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575663.621176:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575663.621178:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575663.621183:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7414 00000100:00080000:2.0:1788575663.621187:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575663.621190:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575663.621195:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575663.621208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff92fc4617f100 x1875449864341504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575663.621209:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864342016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575663.621251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864341504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.621262:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.621267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575663.621271:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864341504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575663.621302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575663.621326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575663.621348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff92fc4617f800 x1875449864341632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.621365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864341632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.621374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.621377:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575663.621380:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575663.621386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575663.621414:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575663.621425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff92fc4617d180 x1875449864341760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.621437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864341760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.621449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.621453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575663.621456:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575663.621461:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575663.621481:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575663.621492:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff92fc4617c380 x1875449864341888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.621503:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864341888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.621510:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.621513:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575663.621516:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864341888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575663.621522:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575663.621548:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575663.621559:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff92fc4617e680 x1875449864342016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.621570:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864342016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.621578:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.621581:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575663.621585:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575664.068866:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575664.132842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575665.092918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575665.156836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575666.116895:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575666.180841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575667.140876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575667.204822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575668.164931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575668.228922:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575668.740853:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575668.740861:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575668.740864:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575668.740872:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7419 00000100:00080000:1.0:1788575668.740879:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575668.740884:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575668.740914:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864342144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575668.740944:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575668.740946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575668.740948:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575668.740950:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7419 00000100:00080000:1.0:1788575668.740952:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575668.740954:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575668.740961:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575668.740969:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864342272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575668.740975:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575668.740978:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575668.740980:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575668.740984:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7419 00000100:00080000:1.0:1788575668.740987:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575668.740990:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864342144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575668.740994:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575668.741003:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575668.741019:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864342400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575668.741030:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575668.741033:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575668.741035:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575668.741039:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7419 00000100:00080000:1.0:1788575668.741043:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575668.741047:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575668.741052:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575668.741060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575668.741064:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864342528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575668.741071:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575668.741073:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575668.741074:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575668.741077:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7419 00000100:00080000:1.0:1788575668.741081:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575668.741083:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575668.741085:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575668.741094:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864342656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575668.741106:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff92fc67288380 x1875449864342144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575668.741142:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864342144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575668.741152:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575668.741156:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575668.741160:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864342144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575668.741167:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575668.741191:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575668.741204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff92fc67289c00 x1875449864342272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575668.741224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864342272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575668.741232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575668.741235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575668.741238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575668.741261:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575668.741280:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575668.741291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff92fc6728a680 x1875449864342400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575668.741302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864342400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575668.741310:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575668.741313:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575668.741316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575668.741321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575668.741338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575668.741349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff92fc67289f80 x1875449864342528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575668.741360:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864342528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575668.741368:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575668.741371:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575668.741374:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575668.741379:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575668.741398:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575668.741408:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff92fc6728b800 x1875449864342656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575668.741419:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864342656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575668.741427:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575668.741430:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575668.741433:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575669.188882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575669.252815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575670.212879:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575670.276843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575671.236864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575671.300787:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575672.260969:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575672.324837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575673.284942:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575673.348871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575673.860898:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575673.860907:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575673.860911:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575673.860921:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7424 00000100:00080000:2.0:1788575673.860931:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575673.860937:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575673.860983:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864342784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575673.861014:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575673.861018:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575673.861019:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575673.861023:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7424 00000100:00080000:2.0:1788575673.861027:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575673.861030:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575673.861038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575673.861051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864342912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575673.861061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575673.861064:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575673.861066:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575673.861072:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7424 00000100:00100000:3.0:1788575673.861072:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864342784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575673.861078:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575673.861083:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575673.861088:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575673.861098:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864343040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575673.861107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575673.861110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575673.861112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575673.861115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7424 00000100:00080000:2.0:1788575673.861141:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575673.861146:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575673.861150:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575673.861156:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575673.861162:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864343168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575673.861177:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575673.861180:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575673.861181:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575673.861185:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7424 00000100:00080000:2.0:1788575673.861188:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575673.861191:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575673.861196:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575673.861204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff92fc4617fb80 x1875449864342784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575673.861208:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864343296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575673.861240:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864342784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575673.861252:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575673.861257:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575673.861261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864342784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575673.861284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575673.861308:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575673.861320:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff92fc4617c000 x1875449864342912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575673.861332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864342912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575673.861340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575673.861343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575673.861347:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864342912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575673.861353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575673.861374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575673.861385:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff92fc4617d880 x1875449864343040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575673.861397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864343040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575673.861405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575673.861408:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575673.861411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575673.861417:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575673.861437:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575673.861448:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff92fc4617d500 x1875449864343168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575673.861460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864343168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575673.861468:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575673.861471:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575673.861474:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575673.861480:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575673.861500:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575673.861511:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575673] req@ffff92fc4617e300 x1875449864343296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575673.861523:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864343296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575673.861530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575673.861534:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575673.861537:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575674.308898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575674.372838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575675.332875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575675.396965:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575676.356869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575676.420860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575677.380876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575677.444860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575678.404840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575678.468877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575678.980900:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.980914:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575678.980919:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.980933:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7429 00000100:00080000:2.0:1788575678.980946:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.980956:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575678.981001:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864343424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.981032:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.981036:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575678.981039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.981043:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7429 00000100:00080000:2.0:1788575678.981046:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.981050:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575678.981062:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575678.981075:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864343552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.981085:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.981088:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575678.981092:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575678.981094:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864343424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575678.981099:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7429 00000100:00080000:2.0:1788575678.981105:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.981110:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575678.981114:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575678.981138:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864343680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.981148:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.981152:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575678.981154:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.981158:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7429 00000100:00080000:2.0:1788575678.981177:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.981182:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575678.981188:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575678.981196:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575678.981201:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864343808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.981211:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.981214:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575678.981217:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.981221:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7429 00000100:00080000:2.0:1788575678.981225:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.981230:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575678.981235:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575678.981248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff92fc4617ce00 x1875449864343424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575678.981249:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864343936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575678.981291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864343424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575678.981305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575678.981310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575678.981316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864343424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575678.981348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575678.981376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575678.981394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff92fc4617ea00 x1875449864343552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575678.981407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864343552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575678.981492:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575678.981497:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575678.981500:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575678.981506:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575678.981634:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575678.981647:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff92fc4617e300 x1875449864343680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575678.981659:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864343680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575678.981709:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575678.981714:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575678.981757:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575678.981766:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575678.981791:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575678.981803:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff92fc4617d500 x1875449864343808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575678.981814:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864343808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575678.981821:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575678.981825:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575678.981895:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575678.981902:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575678.981923:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575678.981935:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff92fc4617d880 x1875449864343936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575678.981946:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864343936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575678.981963:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575678.981966:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575678.981970:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864343936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575679.428914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575679.492890:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575680.453025:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575680.516924:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575681.476894:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575681.540853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575682.500872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575682.564846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575683.524870:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575683.588854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575684.100860:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575684.100870:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575684.100874:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575684.100884:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7434 00000100:00080000:2.0:1788575684.100894:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575684.100901:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575684.100937:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864344064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575684.100965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575684.100968:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575684.100970:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575684.100974:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7434 00000100:00080000:2.0:1788575684.100977:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575684.100980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575684.100988:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575684.101002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864344192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575684.101019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575684.101022:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575684.101024:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575684.101028:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7434 00000100:00080000:2.0:1788575684.101031:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575684.101036:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575684.101046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575684.101046:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864344064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575684.101066:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864344320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575684.101076:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575684.101080:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575684.101081:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575684.101085:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7434 00000100:00080000:2.0:1788575684.101089:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575684.101093:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575684.101098:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575684.101112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864344448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575684.101128:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575684.101131:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575684.101133:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575684.101135:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7434 00000100:00080000:2.0:1788575684.101139:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575684.101142:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575684.101146:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575684.101152:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575684.101156:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864344576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575684.101237:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575684/real 1788575684] req@ffff92fc4617c000 x1875449864344064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575684.101259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864344064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575684.101272:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575684.101278:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575684.101282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864344064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575684.101291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575684.101317:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575684.101332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575684/real 1788575684] req@ffff92fc4617fb80 x1875449864344192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575684.101344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864344192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575684.101353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575684.101356:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575684.101360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575684.101388:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575684.101408:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575684.101419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575684/real 1788575684] req@ffff92fc4617e680 x1875449864344320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575684.101430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864344320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575684.101438:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575684.101442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575684.101445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575684.101451:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575684.101469:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575684.101479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575684/real 1788575684] req@ffff92fc4617c380 x1875449864344448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575684.101490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864344448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575684.101497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575684.101501:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575684.101504:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575684.101510:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575684.101530:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575684.101540:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575684/real 1788575684] req@ffff92fc4617d180 x1875449864344576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575684.101552:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864344576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575684.101559:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575684.101563:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575684.101566:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575684.548913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575684.612844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575685.572916:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575685.638280:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575686.596935:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575686.660854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575687.620967:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575687.684867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575688.644864:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575688.708785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575689.220860:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575689.220869:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575689.220873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575689.220883:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7439 00000100:00080000:0.0:1788575689.220892:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575689.220899:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575689.220932:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864344704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575689.220966:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575689.220968:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575689.220970:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575689.220973:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7439 00000100:00080000:0.0:1788575689.220975:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575689.220977:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575689.220987:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575689.221002:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864344832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575689.221010:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864344704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575689.221015:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575689.221019:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575689.221021:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575689.221026:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7439 00000100:00080000:0.0:1788575689.221030:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575689.221034:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575689.221039:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575689.221051:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864344960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575689.221062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788575689.221062:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575689.221067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575689.221069:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575689.221074:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7439 00000100:00080000:0.0:1788575689.221078:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575689.221081:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575689.221085:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575689.221095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864345088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575689.221103:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575689.221108:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575689.221109:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff92fc67048a80 x1875449864344704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575689.221111:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575689.221128:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7439 00000100:00080000:0.0:1788575689.221133:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575689.221138:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575689.221142:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575689.221147:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864344704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575689.221154:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864345216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575689.221163:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575689.221169:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575689.221174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864344704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575689.221198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575689.221216:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575689.221225:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff92fc67048000 x1875449864344832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575689.221232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864344832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575689.221237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575689.221239:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575689.221241:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575689.221245:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575689.221254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575689.221260:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff92fc67048380 x1875449864344960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575689.221266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864344960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575689.221270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575689.221277:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575689.221278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864344960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575689.221282:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575689.221291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575689.221297:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff92fc67048700 x1875449864345088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575689.221304:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864345088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575689.221308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575689.221310:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575689.221312:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575689.221315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575689.221323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575689.221327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575689/real 1788575689] req@ffff92fc67049f80 x1875449864345216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575689.221333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864345216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575689.221338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575689.221339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575689.221341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575689.668921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575689.732844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575690.692956:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575690.756845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575691.716957:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575691.780834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575692.740928:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575692.804858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575693.764888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575693.828827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575694.340903:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575694.340916:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575694.340921:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575694.340934:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7444 00000100:00080000:1.0:1788575694.340946:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575694.340955:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575694.341014:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864345344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575694.341045:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575694.341049:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575694.341051:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575694.341055:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7444 00000100:00080000:1.0:1788575694.341059:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575694.341063:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575694.341073:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575694.341087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864345472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575694.341097:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575694.341099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864345344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575694.341102:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575694.341106:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575694.341111:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7444 00000100:00080000:1.0:1788575694.341126:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575694.341132:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575694.341137:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575694.341150:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864345600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575694.341159:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575694.341162:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575694.341164:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575694.341167:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7444 00000100:00080000:1.0:1788575694.341179:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575694.341184:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575694.341189:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575694.341192:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575694.341203:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864345728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575694.341213:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575694.341216:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575694.341218:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575694.341222:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7444 00000100:00080000:1.0:1788575694.341226:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575694.341230:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575694.341235:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575694.341247:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864345856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575694.341248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff92fc6728ad80 x1875449864345344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575694.341282:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864345344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575694.341301:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575694.341305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575694.341310:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864345344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575694.341337:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575694.341355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575694.341365:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff92fc67289880 x1875449864345472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575694.341373:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864345472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575694.341379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575694.341381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575694.341383:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575694.341387:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575694.341397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575694.341403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff92fc6728b100 x1875449864345600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575694.341409:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864345600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575694.341414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575694.341415:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575694.341417:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575694.341421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575694.341431:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575694.341436:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff92fc67288000 x1875449864345728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575694.341443:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864345728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575694.341447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575694.341449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575694.341451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575694.341454:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575694.341463:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575694.341468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575694/real 1788575694] req@ffff92fc67288e00 x1875449864345856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575694.341473:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864345856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575694.341477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575694.341479:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575694.341481:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864345856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575694.788909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575694.852897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575695.812917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575695.876893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575696.836898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575696.901080:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575697.861338:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575697.924835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575698.884884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575698.948829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575699.460781:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575699.460790:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575699.460795:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575699.460804:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7450 00000100:00080000:1.0:1788575699.460814:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575699.460821:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575699.460856:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864345984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575699.460880:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575699.460883:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575699.460884:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575699.460887:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7450 00000100:00080000:1.0:1788575699.460890:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575699.460894:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575699.460903:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575699.460915:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864345984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575699.460918:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864346112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575699.460930:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575699.460933:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575699.460935:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575699.460939:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7450 00000100:00080000:1.0:1788575699.460943:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575699.460947:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575699.460952:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575699.460963:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864346240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575699.460973:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575699.460975:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575699.460980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575699.460982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575699.460987:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7450 00000100:00080000:1.0:1788575699.460991:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575699.460995:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575699.460999:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575699.461009:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864346368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575699.461015:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575699.461018:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575699.461019:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575699.461023:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff92fc6728aa00 x1875449864345984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575699.461024:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7450 00000100:00080000:1.0:1788575699.461029:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575699.461033:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575699.461036:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575699.461043:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864346496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575699.461057:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864345984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575699.461067:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575699.461071:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575699.461074:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864345984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575699.461096:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575699.461143:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575699.461153:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff92fc6728b480 x1875449864346112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575699.461161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864346112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575699.461166:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575699.461168:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575699.461170:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575699.461173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575699.461185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575699.461191:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff92fc6728a300 x1875449864346240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575699.461198:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864346240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575699.461206:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575699.461208:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575699.461210:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575699.461212:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575699.461221:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575699.461226:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff92fc67288e00 x1875449864346368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575699.461232:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864346368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575699.461237:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575699.461238:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575699.461240:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575699.461243:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575699.461251:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575699.461256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575699/real 1788575699] req@ffff92fc67288000 x1875449864346496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575699.461262:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864346496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575699.461266:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575699.461267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575699.461270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575699.908831:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575699.972839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575700.932853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575700.996851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575701.956915:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575702.020849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575702.980904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575703.044831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575704.004906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575704.068835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575704.580854:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575704.580864:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575704.580868:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575704.580878:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7455 00000100:00080000:2.0:1788575704.580888:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575704.580895:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575704.580935:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864346624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575704.580960:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575704.580964:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575704.580966:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575704.580970:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7455 00000100:00080000:2.0:1788575704.580974:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575704.580978:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575704.580986:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575704.580997:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864346752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575704.581005:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864346624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575704.581006:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575704.581017:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575704.581018:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575704.581023:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7455 00000100:00080000:2.0:1788575704.581027:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575704.581031:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575704.581036:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575704.581049:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864346880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575704.581058:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575704.581061:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575704.581063:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575704.581066:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7455 00000100:00080000:2.0:1788575704.581070:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575704.581074:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575704.581078:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575704.581084:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575704.581095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864347008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575704.581104:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575704.581107:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575704.581108:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575704.581110:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7455 00000100:00080000:2.0:1788575704.581124:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575704.581128:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575704.581137:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575704.581149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864347136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575704.581154:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff92fc4617f800 x1875449864346624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575704.581177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864346624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575704.581188:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575704.581192:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575704.581196:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864346624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575704.581219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575704.581236:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575704.581244:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff92fc4617f100 x1875449864346752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575704.581252:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864346752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575704.581257:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575704.581259:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575704.581261:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575704.581264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575704.581274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575704.581280:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff92fc4617dc00 x1875449864346880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575704.581287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864346880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575704.581291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575704.581293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575704.581294:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864346880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575704.581297:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575704.581307:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575704.581312:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff92fc4617c700 x1875449864347008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575704.581318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864347008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575704.581322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575704.581324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575704.581325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575704.581328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575704.581337:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575704.581344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff92fc4617ca80 x1875449864347136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575704.581354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864347136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575704.581361:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575704.581364:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575704.581367:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575705.028899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575705.092863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575706.052898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575706.117078:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575707.076882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575707.140908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575708.101550:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575708.164805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575709.124881:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575709.188834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575709.700847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575709.700857:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575709.700861:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575709.700870:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7460 00000100:00080000:2.0:1788575709.700878:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575709.700884:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575709.700917:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864347264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575709.700942:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575709.700946:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575709.700948:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575709.700951:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7460 00000100:00080000:2.0:1788575709.700954:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575709.700957:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575709.700967:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575709.700980:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864347392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575709.700991:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575709.700994:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864347264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575709.700995:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575709.700999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575709.701006:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7460 00000100:00080000:2.0:1788575709.701009:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575709.701012:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575709.701016:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575709.701026:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864347520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575709.701032:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575709.701034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575709.701035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575709.701037:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7460 00000100:00080000:2.0:1788575709.701046:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575709.701048:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575709.701052:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575709.701058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864347648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575709.701062:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575709.701064:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575709.701065:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575709.701067:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7460 00000100:00080000:2.0:1788575709.701068:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575709.701070:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575709.701073:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575709.701078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575709.701082:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864347776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575709.701143:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff92fc4617f480 x1875449864347264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575709.701161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864347264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575709.701170:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575709.701174:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575709.701177:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864347264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575709.701198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575709.701220:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575709.701232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff92fc4617ed80 x1875449864347392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575709.701243:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864347392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575709.701251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575709.701254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575709.701257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575709.701262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575709.701279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575709.701288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff92fc4617ca80 x1875449864347520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575709.701299:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864347520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575709.701305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575709.701308:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575709.701311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575709.701316:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575709.701333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575709.701343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff92fc4617c700 x1875449864347648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575709.701353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864347648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575709.701359:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575709.701362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575709.701365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575709.701370:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575709.701387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575709.701396:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff92fc4617dc00 x1875449864347776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575709.701406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864347776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575709.701413:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575709.701416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575709.701419:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864347776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575710.148892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575710.212837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575711.172869:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575711.236837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575712.196948:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575712.261179:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575713.220851:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575713.288671:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575714.244865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575714.308839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575714.826631:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575714.826642:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575714.826646:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575714.826656:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7465 00000100:00080000:2.0:1788575714.826665:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575714.826672:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575714.826702:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864347904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575714.826744:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575714.826748:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575714.826750:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575714.826754:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7465 00000100:00080000:2.0:1788575714.826758:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575714.826761:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575714.826770:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575714.826782:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864348032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575714.826792:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575714.826795:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575714.826797:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575714.826801:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7465 00000100:00080000:2.0:1788575714.826804:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575714.826808:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575714.826814:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575714.826826:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864347904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575714.826828:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864348160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575714.826842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575714.826845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575714.826847:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575714.826850:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7465 00000100:00080000:2.0:1788575714.826854:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575714.826858:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575714.826863:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575714.826874:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864348288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575714.826883:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575714.826886:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575714.826887:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575714.826898:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7465 00000800:00000100:3.0:1788575714.827004:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575714.827065:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff92fc4617f100 x1875449864347904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575714.827086:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864347904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575714.827098:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575714.827102:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575714.827106:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864347904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575714.827132:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348032:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788575714.827176:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575714.827183:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575714.827189:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575714.827201:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864348416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575714.827396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575714.827411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff92fc4617f800 x1875449864348032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575714.827425:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864348032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575714.827439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575714.827442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575714.827446:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575714.827471:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575714.827493:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575714.827505:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff92fc4617d180 x1875449864348160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575714.827517:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864348160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575714.827526:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575714.827529:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575714.827532:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575714.827538:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575714.827558:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575714.827569:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff92fc4617c380 x1875449864348288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575714.827581:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864348288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575714.827588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575714.827591:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575714.827595:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575714.827613:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575714.827634:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575714.827645:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff92fc4617e680 x1875449864348416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575714.827656:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864348416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575714.827664:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575714.827668:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575714.827670:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575715.268893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575715.333254:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575716.292863:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575716.356852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575717.316817:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575717.380840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575718.340883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575718.404834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575719.364865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575719.428815:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575719.940840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.940854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575719.940858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.940865:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7470 00000100:00080000:2.0:1788575719.940873:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.940883:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575719.940932:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864348544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.940963:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.940967:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575719.940969:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.940973:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7470 00000100:00080000:2.0:1788575719.940976:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.940980:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575719.940989:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575719.941000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864348672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.941010:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.941012:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575719.941014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.941018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7470 00000100:00080000:2.0:1788575719.941021:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.941024:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575719.941028:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575719.941037:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864348800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.941052:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.941055:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575719.941057:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.941061:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7470 00000100:00080000:2.0:1788575719.941064:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.941067:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575719.941071:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575719.941080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864348928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.941088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.941091:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575719.941092:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.941096:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7470 00000100:00080000:2.0:1788575719.941099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.941102:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575719.941106:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575719.941114:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864349056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575719.941198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864348544:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575719.941250:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575719.941294:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff92fc4617fb80 x1875449864348544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575719.941310:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864348544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575719.941320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575719.941325:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575719.941328:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864348544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575719.941335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575719.941355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575719.941366:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff92fc4617c000 x1875449864348672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575719.941377:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864348672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575719.941384:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575719.941388:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575719.941391:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575719.941396:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575719.941413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575719.941423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff92fc4617d880 x1875449864348800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575719.941441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864348800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575719.941449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575719.941452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575719.941455:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575719.941460:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575719.941478:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575719.941488:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff92fc4617d500 x1875449864348928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575719.941499:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864348928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575719.941507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575719.941510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575719.941513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864348928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575719.941518:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575719.941536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575719.941546:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff92fc4617e300 x1875449864349056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575719.941557:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864349056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575719.941565:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575719.941568:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575719.941570:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575720.388865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575720.452837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575721.413016:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575721.476901:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575722.436972:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575722.500891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575723.460917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575723.524819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575724.484884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575724.548869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575725.060830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575725.060839:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575725.060843:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575725.060853:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7475 00000100:00080000:2.0:1788575725.060864:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575725.060870:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575725.060905:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864349184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575725.060941:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575725.060945:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575725.060950:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575725.060954:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7475 00000100:00080000:2.0:1788575725.060958:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575725.060961:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575725.060971:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575725.060984:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864349312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575725.060987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864349184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575725.061002:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575725.061006:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575725.061008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575725.061013:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7475 00000100:00080000:2.0:1788575725.061017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575725.061021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575725.061025:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575725.061037:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864349440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575725.061047:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575725.061051:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575725.061060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575725.061064:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7475 00000100:00080000:2.0:1788575725.061068:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575725.061073:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575725.061076:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349312:192.168.202.132@tcp:38:kworker.0 00000008:00100000:2.0:1788575725.061081:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575725.061096:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864349568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575725.061099:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349440:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788575725.061115:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575725.061119:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575725.061121:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575725.061126:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349568:192.168.202.132@tcp:8:kworker.0 00000100:00080000:2.0:1788575725.061128:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7475 00000100:00080000:2.0:1788575725.061134:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575725.061139:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575725.061144:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575725.061164:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864349696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575725.061194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575725.508853:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575725.572785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575726.532878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575726.596822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575727.556886:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575727.620818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:3.0:1788575728.133103:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:3.0:1788575728.133155:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575728.133171:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575728.133208:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575728.133217:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575728.133228:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:3.0:1788575728.133237:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575728.133248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575725/real 1788575728] req@ffff92fc4617d500 x1875449864349568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575728.133268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864349568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575728.133284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575728.133289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575728.133293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575728.133300:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575725/real 1788575728] req@ffff92fc4617ce00 x1875449864349312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575728.133313:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864349312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575728.133318:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575728.133320:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575728.133322:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575728.133326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575725/real 1788575728] req@ffff92fc4617e300 x1875449864349440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575728.133333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864349440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575728.133337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575728.133339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575728.133341:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575728.133344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575725/real 1788575728] req@ffff92fc4617ea00 x1875449864349184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575728.133350:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864349184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575728.133356:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575728.133358:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575728.133360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864349184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575728.133378:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575725/real 1788575728] req@ffff92fc4617d880 x1875449864349696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575728.133384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864349696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575728.133389:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575728.133391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575728.133392:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575728.580924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575728.644879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575729.604874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575729.668820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575730.181331:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575730.181342:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575730.181346:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575730.181357:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7480 00000100:00080000:1.0:1788575730.181367:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575730.181375:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575730.181422:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864349824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575730.181459:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575730.181464:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575730.181466:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575730.181469:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7480 00000100:00080000:1.0:1788575730.181474:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575730.181478:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575730.181487:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575730.181499:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864349952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575730.181507:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575730.181510:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575730.181512:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575730.181515:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7480 00000100:00080000:1.0:1788575730.181518:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575730.181520:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575730.181524:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575730.181533:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864350080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575730.181541:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575730.181544:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575730.181546:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575730.181549:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7480 00000100:00080000:1.0:1788575730.181552:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575730.181555:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575730.181559:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575730.181567:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864350208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575730.181587:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575730.181591:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575730.181592:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575730.181596:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7480 00000100:00080000:1.0:1788575730.181599:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575730.181602:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575730.181607:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575730.181617:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864350336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575730.182177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864349824:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575730.182237:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575730.182626:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff92fc6728b100 x1875449864349824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575730.182646:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864349824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575730.182656:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575730.182660:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575730.182663:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864349824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575730.182670:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575730.182695:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575730.182707:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff92fc67289880 x1875449864349952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575730.182718:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864349952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575730.182754:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575730.182757:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575730.182760:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864349952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575730.182767:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575730.182786:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575730.182796:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff92fc6728ad80 x1875449864350080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575730.182805:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864350080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575730.182811:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575730.182814:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575730.182817:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575730.182821:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575730.182838:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575730.182848:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff92fc6728b800 x1875449864350208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575730.182858:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864350208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575730.182865:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575730.182867:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575730.182870:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575730.182876:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575730.182895:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575730.182906:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575730/real 1788575730] req@ffff92fc67289f80 x1875449864350336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575730.182917:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864350336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575730.182924:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575730.182926:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575730.182929:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575730.630569:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575730.692835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575731.654462:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575731.716863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575732.676851:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575732.740838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575733.701702:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575733.765147:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575734.724801:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575734.788834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575735.300839:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575735.300849:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575735.300853:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575735.300863:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7485 00000100:00080000:1.0:1788575735.300873:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575735.300882:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575735.300920:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864350464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575735.300946:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575735.300950:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575735.300952:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575735.300956:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7485 00000100:00080000:1.0:1788575735.300960:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575735.300964:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575735.300974:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575735.300988:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864350592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575735.300996:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864350464:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575735.301002:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575735.301006:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575735.301008:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575735.301012:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7485 00000100:00080000:1.0:1788575735.301015:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575735.301019:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575735.301024:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575735.301037:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864350720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575735.301046:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575735.301049:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575735.301050:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575735.301053:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7485 00000100:00080000:1.0:1788575735.301056:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575735.301060:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575735.301061:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575735.301067:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575735.301075:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864350848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575735.301081:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575735.301084:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575735.301085:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575735.301087:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7485 00000100:00080000:1.0:1788575735.301090:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575735.301093:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575735.301098:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575735.301105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff92fc6728a680 x1875449864350464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575735.301110:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864350976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575735.301136:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864350464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575735.301147:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575735.301151:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575735.301168:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864350464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575735.301190:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575735.301212:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575735.301224:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff92fc67289c00 x1875449864350592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575735.301234:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864350592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575735.301248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575735.301251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575735.301254:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575735.301259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575735.301274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575735.301283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff92fc67288380 x1875449864350720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575735.301293:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864350720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575735.301300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575735.301303:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575735.301305:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575735.301310:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575735.301326:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575735.301334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff92fc6728bb80 x1875449864350848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575735.301344:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864350848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575735.301352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575735.301355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575735.301358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575735.301364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575735.301383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575735.301394:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575735/real 1788575735] req@ffff92fc67288700 x1875449864350976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575735.301403:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864350976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575735.301410:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575735.301413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575735.301416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864350976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575735.748876:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575735.812798:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575736.772892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575736.836842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575737.797213:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575737.860838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575738.820831:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575738.884835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575739.844825:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575739.908833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575740.420859:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575740.420869:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575740.420873:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575740.420894:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7491 00000100:00080000:1.0:1788575740.420916:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575740.420924:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575740.420973:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864351104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575740.420999:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575740.421003:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575740.421005:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575740.421009:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7491 00000100:00080000:1.0:1788575740.421014:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575740.421017:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575740.421026:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575740.421039:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864351232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575740.421043:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864351104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575740.421064:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575740.421067:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575740.421069:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575740.421073:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7491 00000100:00080000:1.0:1788575740.421077:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575740.421081:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575740.421086:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575740.421099:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864351360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575740.421108:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575740.421109:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575740.421114:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575740.421117:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575740.421122:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7491 00000100:00080000:1.0:1788575740.421127:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575740.421131:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575740.421137:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575740.421145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff92fc67289180 x1875449864351104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575740.421151:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864351488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575740.421175:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575740.421177:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575740.421179:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575740.421182:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7491 00000100:00100000:3.0:1788575740.421186:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864351104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575740.421187:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575740.421193:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575740.421199:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575740.421210:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864351616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575740.421210:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575740.421218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575740.421224:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864351104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575740.421246:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575740.421276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575740.421289:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff92fc67289500 x1875449864351232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575740.421301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864351232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575740.421309:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575740.421312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575740.421316:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575740.421321:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575740.421340:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575740.421351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff92fc67288a80 x1875449864351360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575740.421361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864351360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575740.421366:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575740.421368:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575740.421370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575740.421373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575740.421386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575740.421393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff92fc67288700 x1875449864351488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575740.421399:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864351488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575740.421404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575740.421406:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575740.421408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575740.421411:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575740.421423:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575740.421432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575740/real 1788575740] req@ffff92fc6728bb80 x1875449864351616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575740.421442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864351616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575740.421449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575740.421452:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575740.421454:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575740.868850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575740.932817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575741.892873:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575741.956814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575742.916878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575742.980858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575743.940875:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575744.004855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575744.964876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575745.028825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575745.540824:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575745.540834:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575745.540838:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575745.540846:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7496 00000100:00080000:1.0:1788575745.540866:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575745.540873:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575745.540913:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864351744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575745.540942:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575745.540946:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575745.540948:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575745.540951:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7496 00000100:00080000:1.0:1788575745.540955:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575745.540958:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575745.540967:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575745.540978:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864351872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575745.540988:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575745.540993:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575745.540995:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575745.540998:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7496 00000100:00080000:1.0:1788575745.541001:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575745.541004:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864351744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575745.541007:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575745.541016:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575745.541027:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864352000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575745.541036:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575745.541039:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575745.541041:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575745.541045:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7496 00000100:00080000:1.0:1788575745.541048:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575745.541051:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575745.541056:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575745.541066:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864352128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575745.541072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575745.541080:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575745.541083:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575745.541085:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575745.541090:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7496 00000100:00080000:1.0:1788575745.541094:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575745.541097:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575745.541102:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575745.541113:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864352256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575745.541117:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff92fc67288380 x1875449864351744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575745.541143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864351744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575745.541160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575745.541164:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575745.541167:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864351744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575745.541177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575745.541197:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575745.541208:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff92fc67289c00 x1875449864351872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575745.541218:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864351872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575745.541225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575745.541228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575745.541230:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864351872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575745.541253:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575745.541268:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575745.541278:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff92fc6728a680 x1875449864352000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575745.541288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864352000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575745.541296:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575745.541299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575745.541302:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575745.541307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575745.541325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575745.541335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff92fc67289f80 x1875449864352128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575745.541346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864352128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575745.541352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575745.541355:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575745.541358:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575745.541363:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575745.541379:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575745.541388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575745/real 1788575745] req@ffff92fc6728b800 x1875449864352256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575745.541398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864352256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575745.541406:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575745.541409:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575745.541412:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575745.988898:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575746.053754:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575747.012922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575747.076785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575748.036874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575748.100819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575749.060884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575749.124840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575750.084857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575750.148843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575750.660865:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.660875:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575750.660879:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.660888:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7501 00000100:00080000:2.0:1788575750.660908:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.660917:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575750.660952:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864352384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575750.660980:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.660984:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575750.660986:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.660991:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7501 00000100:00080000:2.0:1788575750.660994:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.660998:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575750.661020:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575750.661022:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864352384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575750.661037:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864352512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575750.661050:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.661053:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575750.661056:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.661061:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7501 00000100:00080000:2.0:1788575750.661065:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.661070:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575750.661075:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575750.661090:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575750.661097:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864352640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575750.661110:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.661113:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575750.661115:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.661119:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7501 00000100:00080000:2.0:1788575750.661124:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.661130:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575750.661132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff92fc4617c000 x1875449864352384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575750.661146:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575750.661153:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864352384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.661160:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864352768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575750.661169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.661173:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575750.661176:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788575750.661178:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575750.661182:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575750.661184:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864352384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575750.661188:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7501 00000100:00080000:2.0:1788575750.661193:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.661197:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575750.661202:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575750.661208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575750.661214:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864352896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575750.661234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575750.661246:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff92fc4617fb80 x1875449864352512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.661254:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864352512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.661260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.661262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575750.661264:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575750.661267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575750.661279:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575750.661284:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff92fc4617e680 x1875449864352640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.661291:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864352640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.661295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.661296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575750.661298:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575750.661301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575750.661321:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575750.661332:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff92fc4617c380 x1875449864352768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.661345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864352768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.661352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.661357:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575750.661360:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575750.661379:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575750.661424:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575750.661446:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff92fc4617d180 x1875449864352896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575750.661460:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864352896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575750.661466:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575750.661468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575750.661470:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864352896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575751.108864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575751.172824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575752.132915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575752.196827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575753.156876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575753.220856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575754.180857:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575754.244823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575755.204878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575755.268848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575755.780826:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575755.780836:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575755.780839:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575755.780848:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7506 00000100:00080000:1.0:1788575755.780858:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575755.780866:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575755.780917:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864353024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575755.780963:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575755.780967:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575755.780969:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575755.780973:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7506 00000100:00080000:1.0:1788575755.780977:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575755.780981:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575755.780992:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575755.781004:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864353024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575755.781009:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864353152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575755.781029:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575755.781033:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575755.781035:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575755.781038:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7506 00000100:00080000:1.0:1788575755.781041:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575755.781045:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575755.781050:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575755.781063:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864353280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575755.781072:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575755.781076:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575755.781079:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575755.781083:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7506 00000100:00080000:1.0:1788575755.781087:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575755.781092:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788575755.781097:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575755.781099:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575755.781112:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864353408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575755.781122:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575755.781125:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575755.781127:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575755.781131:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7506 00000100:00080000:1.0:1788575755.781136:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575755.781140:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575755.781145:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575755.781151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff92fc6728ad80 x1875449864353024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575755.781171:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864353536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575755.781185:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864353024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575755.781199:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575755.781202:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575755.781205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864353024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575755.781228:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575755.781256:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575755.781268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff92fc67289880 x1875449864353152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575755.781280:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864353152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575755.781289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575755.781292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575755.781296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575755.781300:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575755.781314:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575755.781321:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff92fc6728b100 x1875449864353280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575755.781329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864353280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575755.781334:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575755.781337:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575755.781338:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575755.781342:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575755.781353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575755.781359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff92fc67288000 x1875449864353408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575755.781366:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864353408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575755.781371:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575755.781373:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575755.781375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575755.781378:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575755.781386:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575755.781391:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff92fc67288e00 x1875449864353536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575755.781397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864353536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575755.781402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575755.781404:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575755.781405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575756.228925:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575756.292929:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575757.252931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575757.316814:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575758.276911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575758.340827:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575759.300846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575759.364821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575760.324865:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575760.388843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575760.900841:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.900851:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575760.900856:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.900866:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7511 00000100:00080000:3.0:1788575760.900876:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.900882:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575760.900926:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864353664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575760.900953:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.900956:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575760.900964:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.900970:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7511 00000100:00080000:3.0:1788575760.900975:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.900979:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575760.900986:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864353664:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788575760.900990:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575760.901006:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864353792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575760.901017:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.901020:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575760.901021:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.901024:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7511 00000100:00080000:3.0:1788575760.901028:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.901030:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575760.901033:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575760.901042:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864353920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575760.901050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575760.901063:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.901065:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575760.901067:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.901070:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7511 00000100:00080000:3.0:1788575760.901072:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.901075:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575760.901077:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575760.901085:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864354048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575760.901091:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.901093:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575760.901096:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575760.901097:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff92fc67339180 x1875449864353664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575760.901099:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7511 00000100:00080000:3.0:1788575760.901103:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.901106:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575760.901112:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575760.901119:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864354176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575760.901137:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864353664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.901149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.901154:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575760.901157:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864353664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575760.901178:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575760.901195:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575760.901204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff92fc6733ad80 x1875449864353792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.901217:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864353792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.901223:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.901225:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575760.901228:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575760.901231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575760.901242:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575760.901248:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff92fc67339f80 x1875449864353920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.901255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864353920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.901260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.901261:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575760.901263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864353920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575760.901266:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575760.901276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575760.901281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff92fc67339c00 x1875449864354048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.901287:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864354048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.901291:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.901293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575760.901295:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575760.901297:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575760.901306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575760.901311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff92fc67339880 x1875449864354176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575760.901318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864354176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575760.901322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575760.901323:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575760.901325:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575761.348880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575761.412836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575762.372856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575762.436854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575763.396864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575763.460848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575764.420919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575764.484819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575765.444876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575765.508833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575766.020817:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575766.020825:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575766.020828:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575766.020836:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7516 00000100:00080000:3.0:1788575766.020845:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575766.020850:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575766.020878:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864354304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575766.020901:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575766.020904:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575766.020905:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575766.020907:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7516 00000100:00080000:3.0:1788575766.020909:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575766.020911:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575766.020918:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575766.020927:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864354432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575766.020934:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575766.020936:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575766.020937:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575766.020940:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7516 00000100:00080000:3.0:1788575766.020941:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575766.020944:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575766.020947:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864354304:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788575766.020948:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575766.020957:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864354560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575766.020965:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575766.020969:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575766.020971:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575766.020974:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7516 00000100:00080000:3.0:1788575766.020977:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575766.020980:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575766.020986:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575766.020996:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864354688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575766.021005:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575766.021010:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575766.021011:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575766.021015:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575766.021019:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7516 00000100:00080000:3.0:1788575766.021022:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575766.021025:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575766.021029:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575766.021038:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864354816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575766.021054:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575766/real 1788575766] req@ffff92fc6733b800 x1875449864354304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575766.021068:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864354304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575766.021074:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575766.021077:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575766.021079:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864354304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575766.021083:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575766.021098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575766.021106:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575766/real 1788575766] req@ffff92fc67338a80 x1875449864354432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575766.021113:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864354432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575766.021122:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575766.021124:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575766.021126:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575766.021143:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575766.021153:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575766.021159:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575766/real 1788575766] req@ffff92fc67338380 x1875449864354560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575766.021165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864354560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575766.021169:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575766.021171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575766.021173:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575766.021176:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575766.021185:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575766.021189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575766/real 1788575766] req@ffff92fc67338000 x1875449864354688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575766.021195:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864354688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575766.021200:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575766.021201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575766.021203:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575766.021206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575766.021215:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575766.021219:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575766/real 1788575766] req@ffff92fc6733aa00 x1875449864354816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575766.021225:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864354816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575766.021230:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575766.021231:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575766.021233:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864354816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575766.471792:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575766.532822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575767.492843:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575767.556834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575768.516872:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575768.580813:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575769.540840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575769.604819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575770.564835:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575770.628823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575771.140862:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575771.140883:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575771.140888:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575771.140898:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7521 00000100:00080000:3.0:1788575771.140910:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575771.140917:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575771.140962:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864354944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575771.141000:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575771.141003:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575771.141005:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575771.141008:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7521 00000100:00080000:3.0:1788575771.141011:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575771.141014:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575771.141024:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575771.141034:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864355072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575771.141041:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575771.141044:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575771.141045:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575771.141048:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7521 00000100:00080000:3.0:1788575771.141051:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575771.141055:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575771.141060:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575771.141070:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864355200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575771.141085:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864354944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575771.141086:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575771.141102:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575771.141105:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575771.141110:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7521 00000100:00080000:3.0:1788575771.141114:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575771.141118:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575771.141125:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575771.141140:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864355328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575771.141152:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575771.141157:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788575771.141158:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575771.141165:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575771.141170:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7521 00000100:00080000:3.0:1788575771.141175:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575771.141179:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575771.141184:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575771.141195:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864355456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575771.141205:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575771/real 1788575771] req@ffff92fc67339500 x1875449864354944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575771.141229:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864354944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575771.141240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575771.141246:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575771.141250:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864354944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575771.141257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575771.141278:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575771.141291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575771/real 1788575771] req@ffff92fc6733a300 x1875449864355072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575771.141301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864355072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575771.141308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575771.141311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575771.141314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575771.141338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575771.141355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575771.141366:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575771/real 1788575771] req@ffff92fc6733b480 x1875449864355200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575771.141375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864355200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575771.141382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575771.141385:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575771.141387:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575771.141393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575771.141422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575771.141432:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575771/real 1788575771] req@ffff92fc67338700 x1875449864355328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575771.141442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864355328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575771.141447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575771.141449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575771.141451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575771.141455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575771.141468:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575771.141474:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575771/real 1788575771] req@ffff92fc6733bb80 x1875449864355456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575771.141481:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864355456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575771.141485:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575771.141487:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575771.141488:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575771.589438:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575771.652825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575772.612831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575772.676820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575773.636794:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575773.700840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575774.660831:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575774.724839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575775.684857:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575775.748852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575776.260846:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575776.260858:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575776.260862:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575776.260873:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7526 00000100:00080000:3.0:1788575776.260882:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575776.260890:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575776.260924:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864355584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575776.260950:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575776.260957:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575776.260958:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575776.260962:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7526 00000100:00080000:3.0:1788575776.260965:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575776.260969:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575776.260979:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575776.260991:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864355712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575776.261002:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575776.261006:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575776.261007:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864355584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575776.261009:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575776.261017:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7526 00000100:00080000:3.0:1788575776.261020:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575776.261023:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575776.261028:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575776.261039:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864355840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575776.261047:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575776.261051:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575776.261053:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575776.261057:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7526 00000100:00080000:3.0:1788575776.261060:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575776.261064:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575776.261070:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575776.261075:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575776.261082:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864355968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575776.261093:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575776.261096:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575776.261097:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575776.261101:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7526 00000100:00080000:3.0:1788575776.261104:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575776.261108:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575776.261113:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575776.261119:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff92fc67338e00 x1875449864355584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575776.261126:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864356096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575776.261148:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864355584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.261185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.261189:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575776.261193:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864355584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575776.261226:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575776.261249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575776.261262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff92fc6733a680 x1875449864355712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.261273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864355712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.261281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.261284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575776.261287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575776.261293:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575776.261310:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575776.261319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff92fc6733b100 x1875449864355840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.261330:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864355840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.261337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.261340:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575776.261343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575776.261348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575776.261366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575776.261377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff92fc6733bb80 x1875449864355968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.261388:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864355968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.261395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.261398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575776.261400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864355968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575776.261405:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575776.261421:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575776.261430:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575776/real 1788575776] req@ffff92fc67338700 x1875449864356096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575776.261441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864356096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575776.261447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575776.261450:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575776.261452:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575776.708829:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575776.772866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575777.732867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575777.796835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575778.756890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575778.820808:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575779.780911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575779.844854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575780.804914:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575780.868888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575781.380859:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.380869:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575781.380873:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.380883:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7531 00000100:00080000:0.0:1788575781.380893:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.380900:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575781.380974:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864356224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.381009:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.381013:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575781.381014:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.381018:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7531 00000100:00080000:0.0:1788575781.381021:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.381024:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575781.381033:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575781.381045:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864356352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.381056:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.381059:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575781.381060:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.381064:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7531 00000100:00080000:0.0:1788575781.381067:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575781.381071:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864356224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575781.381074:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575781.381083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575781.381095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864356480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.381113:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.381116:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575781.381118:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.381122:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7531 00000100:00080000:0.0:1788575781.381125:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.381128:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:1.0:1788575781.381133:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:0.0:1788575781.381134:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575781.381148:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864356608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575781.381156:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575781.381159:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575781.381160:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575781.381163:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7531 00000100:00080000:0.0:1788575781.381166:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575781.381168:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575781.381173:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575781.381176:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff92fc6704a300 x1875449864356224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575781.381183:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864356736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575781.381199:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864356224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.381215:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.381218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575781.381223:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864356224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575781.381229:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575781.381246:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575781.381256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff92fc6704aa00 x1875449864356352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.381266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864356352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.381273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.381276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575781.381279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575781.381302:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575781.381316:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575781.381324:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff92fc6704a680 x1875449864356480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.381333:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864356480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.381340:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.381342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575781.381345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575781.381350:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575781.381364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575781.381372:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff92fc6704ad80 x1875449864356608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.381381:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864356608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.381388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.381390:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575781.381393:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575781.381397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575781.381411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575781.381433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575781/real 1788575781] req@ffff92fc67049880 x1875449864356736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575781.381442:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864356736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575781.381449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575781.381451:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575781.381454:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575781.828824:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575781.892818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575782.852860:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575782.916829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575783.876846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575783.940843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575784.900856:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575784.964859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575785.924878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575785.988830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575786.500890:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575786.500901:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575786.500906:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575786.500916:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7537 00000100:00080000:2.0:1788575786.500926:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575786.500934:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575786.500968:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864356864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575786.500998:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575786.501005:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575786.501007:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575786.501012:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7537 00000100:00080000:2.0:1788575786.501017:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575786.501021:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575786.501030:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864356864:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788575786.501031:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575786.501049:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864356992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575786.501058:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575786.501061:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575786.501062:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575786.501065:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7537 00000100:00080000:2.0:1788575786.501069:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575786.501071:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575786.501075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575786.501083:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864357120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575786.501091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575786.501093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575786.501094:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575786.501095:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575786.501097:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7537 00000100:00080000:2.0:1788575786.501101:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575786.501103:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575786.501107:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575786.501112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864357248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575786.501117:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575786.501119:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575786.501120:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575786.501122:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7537 00000100:00080000:2.0:1788575786.501124:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575786.501127:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575786.501130:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575786.501136:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864357376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575786.501145:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff92fc4617f800 x1875449864356864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.501168:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864356864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.501178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.501182:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575786.501186:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864356864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575786.501208:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575786.501230:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575786.501240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff92fc4617f100 x1875449864356992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.501247:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864356992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.501253:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.501255:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575786.501258:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864356992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575786.501262:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575786.501275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575786.501282:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff92fc4617dc00 x1875449864357120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.501290:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864357120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.501295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.501297:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575786.501300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575786.501304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575786.501315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575786.501323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff92fc4617c700 x1875449864357248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.501332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864357248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.501337:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.501340:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575786.501342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575786.501347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575786.501370:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575786.501379:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575786/real 1788575786] req@ffff92fc4617ca80 x1875449864357376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575786.501386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864357376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575786.501391:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575786.501393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575786.501394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575786.948877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575787.012895:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575787.972898:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575788.036829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575788.996884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575789.060829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575790.020920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575790.084857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575791.044884:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575791.108802:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575791.620847:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.620858:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575791.620863:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.620872:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7542 00000100:00080000:2.0:1788575791.620882:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.620889:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575791.620923:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864357504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.620948:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.620952:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.620954:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.620958:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7542 00000100:00080000:2.0:1788575791.620961:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.620965:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575791.620975:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575791.620988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864357632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575791.620992:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864357504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575791.621004:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.621007:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575791.621010:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.621014:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7542 00000100:00080000:2.0:1788575791.621018:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.621022:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575791.621027:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575791.621040:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864357760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.621048:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.621052:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.621054:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.621057:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7542 00000100:00080000:2.0:1788575791.621061:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.621064:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575791.621068:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575791.621077:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864357888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.621088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.621090:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575791.621091:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.621095:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7542 00000100:00100000:0.0:1788575791.621098:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357632:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788575791.621099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.621105:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575791.621109:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575791.621119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864358016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575791.621137:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575791.621162:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575791.621179:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575792.068881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575792.132824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575793.092859:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575793.156856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575794.116955:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575794.180892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788575794.693140:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788575794.693197:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575794.693218:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575794.693264:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575794.693278:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575794.693293:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575794.693295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff92fc4617f480 x1875449864357632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788575794.693307:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575794.693337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864357632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575794.693352:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575794.693359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575794.693364:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575794.693373:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff92fc4617ca80 x1875449864357760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575794.693387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864357760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575794.693395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575794.693400:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575794.693404:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575794.693411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff92fc4617c700 x1875449864357888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575794.693424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864357888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575794.693433:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575794.693437:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575794.693441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864357888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575794.693449:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff92fc4617ed80 x1875449864357504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575794.693463:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864357504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575794.693472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575794.693475:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575794.693479:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864357504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575794.693515:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575794] req@ffff92fc4617dc00 x1875449864358016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575794.693529:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864358016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575794.693537:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575794.693541:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575794.693544:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575795.140826:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575795.204881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575796.164900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575796.228857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575796.740876:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575796.740891:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575796.740896:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575796.740909:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7547 00000100:00080000:0.0:1788575796.740922:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575796.740931:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575796.740978:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864358144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575796.741017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575796.741021:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575796.741023:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575796.741028:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7547 00000100:00080000:0.0:1788575796.741032:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575796.741037:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575796.741049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575796.741065:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864358272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575796.741077:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864358144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575796.741087:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575796.741092:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575796.741094:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575796.741099:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7547 00000100:00080000:0.0:1788575796.741103:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575796.741107:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575796.741118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575796.741134:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864358400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575796.741144:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575796.741148:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575796.741150:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575796.741160:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7547 00000800:00000100:1.0:1788575796.741166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575796.741167:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575796.741174:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575796.741179:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575796.741193:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864358528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575796.741203:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575796.741207:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575796.741208:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575796.741213:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7547 00000100:00100000:1.0:1788575796.741218:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff92fc67049c00 x1875449864358144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.741219:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575796.741234:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575796.741240:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575796.741246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864358144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.741256:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864358656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575796.741260:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575796.741269:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575796.741282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864358144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575796.741317:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575796.741344:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575796.741358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff92fc67048e00 x1875449864358272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575796.741369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864358272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575796.741377:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575796.741381:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575796.741384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575796.741390:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575796.741410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575796.741420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff92fc67049180 x1875449864358400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575796.741430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864358400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575796.741437:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575796.741441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575796.741443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575796.741449:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575796.741467:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575796.741476:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff92fc6704bb80 x1875449864358528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575796.741494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864358528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575796.741511:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575796.741515:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575796.741517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575796.741522:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575796.741542:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575796.741551:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff92fc67049500 x1875449864358656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575796.741562:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864358656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575796.741568:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575796.741571:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575796.741573:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575797.188877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575797.252839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575798.212900:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575798.276826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575799.236891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575799.300818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575800.260849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575800.324816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575801.284874:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575801.348825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575801.860811:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575801.860819:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575801.860822:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575801.860830:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7552 00000100:00080000:3.0:1788575801.860838:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575801.860843:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575801.860871:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864358784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575801.860890:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575801.860893:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575801.860894:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575801.860896:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7552 00000100:00080000:3.0:1788575801.860899:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575801.860902:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575801.860909:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575801.860918:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864358912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575801.860922:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864358784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575801.860929:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575801.860931:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575801.860932:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575801.860936:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7552 00000100:00080000:3.0:1788575801.860939:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575801.860943:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575801.860947:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575801.860960:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864359040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575801.860970:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575801.860973:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575801.860975:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575801.860978:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7552 00000100:00080000:3.0:1788575801.860982:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575801.860986:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575801.860992:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575801.860996:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575801.861006:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864359168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575801.861022:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575801.861025:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575801.861027:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575801.861036:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff92fc6733b480 x1875449864358784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575801.861040:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7552 00000100:00080000:3.0:1788575801.861048:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575801.861055:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575801.861064:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575801.861070:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864358784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575801.861077:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864359296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575801.861078:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.861090:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575801.861093:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864358784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575801.861112:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575801.861128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575801.861136:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff92fc6733a300 x1875449864358912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.861143:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864358912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.861149:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.861151:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575801.861152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864358912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575801.861155:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575801.861165:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575801.861170:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff92fc67339500 x1875449864359040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.861177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864359040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.861185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.861187:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575801.861189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575801.861192:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575801.861201:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575801.861206:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff92fc6733aa00 x1875449864359168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.861212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864359168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.861216:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.861218:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575801.861219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575801.861222:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575801.861231:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575801.861235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff92fc67338000 x1875449864359296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575801.861242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864359296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575801.861246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575801.861247:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575801.861249:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575802.308861:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575802.372833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575803.332891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575803.396834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575804.356907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575804.420861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575805.380917:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575805.444829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575806.404849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575806.468803:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575806.980845:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575806.980855:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575806.980859:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575806.980869:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7557 00000100:00080000:1.0:1788575806.980879:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575806.980886:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575806.980927:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864359424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575806.980954:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575806.980957:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575806.980958:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575806.980961:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7557 00000100:00080000:1.0:1788575806.980965:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575806.980969:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575806.980978:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575806.980991:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864359552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575806.980996:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864359424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575806.981010:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575806.981012:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575806.981013:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575806.981017:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7557 00000100:00080000:1.0:1788575806.981020:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575806.981022:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575806.981026:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575806.981035:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864359680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575806.981041:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575806.981043:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575806.981044:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575806.981046:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7557 00000100:00080000:1.0:1788575806.981048:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575806.981051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575806.981052:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575806.981057:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575806.981066:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864359808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575806.981071:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575806.981073:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575806.981074:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575806.981076:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7557 00000100:00080000:1.0:1788575806.981079:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575806.981081:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575806.981084:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575806.981087:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff92fc6728a300 x1875449864359424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575806.981094:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864359936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575806.981104:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864359424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575806.981114:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575806.981117:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575806.981120:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864359424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575806.981137:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575806.981149:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575806.981156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff92fc6728b480 x1875449864359552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575806.981163:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864359552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575806.981168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575806.981170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575806.981171:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575806.981175:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575806.981184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575806.981189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff92fc6728aa00 x1875449864359680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575806.981195:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864359680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575806.981200:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575806.981201:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575806.981203:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575806.981205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575806.981232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575806.981239:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff92fc67288e00 x1875449864359808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575806.981246:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864359808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575806.981251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575806.981253:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575806.981255:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575806.981258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575806.981267:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575806.981271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575806] req@ffff92fc67288000 x1875449864359936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575806.981278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864359936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575806.981282:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575806.981284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575806.981285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864359936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575807.428929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575807.492841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575808.452854:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575808.516825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575809.476919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575809.540844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575810.500855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575810.564820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575811.524894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575811.588863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575812.100793:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575812.100800:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575812.100803:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575812.100811:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7562 00000100:00080000:3.0:1788575812.100819:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575812.100824:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575812.100851:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864360064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575812.100873:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575812.100876:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575812.100878:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575812.100881:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7562 00000100:00080000:3.0:1788575812.100885:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575812.100889:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575812.100899:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575812.100913:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864360192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575812.100918:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864360064:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575812.100925:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575812.100928:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575812.100929:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575812.100932:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7562 00000100:00080000:3.0:1788575812.100936:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575812.100939:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575812.100943:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575812.100954:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864360320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575812.100963:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575812.100966:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575812.100968:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575812.100973:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7562 00000800:00000100:0.0:1788575812.100974:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575812.100979:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575812.100984:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575812.100990:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575812.101002:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864360448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575812.101012:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575812.101015:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575812.101017:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575812.101022:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7562 00000100:00080000:3.0:1788575812.101025:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575812.101027:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575812/real 1788575812] req@ffff92fc67338380 x1875449864360064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575812.101034:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575812.101042:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575812.101046:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864360064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575812.101056:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575812.101057:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864360576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575812.101062:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575812.101078:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864360064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575812.101103:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575812.101128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575812.101139:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575812/real 1788575812] req@ffff92fc67338a80 x1875449864360192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575812.101152:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864360192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575812.101160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575812.101163:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575812.101165:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575812.101169:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575812.101183:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575812.101189:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575812/real 1788575812] req@ffff92fc6733b800 x1875449864360320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575812.101197:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864360320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575812.101202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575812.101203:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575812.101205:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575812.101215:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575812.101225:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575812.101231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575812/real 1788575812] req@ffff92fc67339880 x1875449864360448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575812.101241:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864360448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575812.101248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575812.101251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575812.101253:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575812.101259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575812.101276:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575812.101286:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575812/real 1788575812] req@ffff92fc67339c00 x1875449864360576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575812.101296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864360576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575812.101305:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575812.101307:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575812.101309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575812.548819:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575812.612854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575813.572944:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575813.636843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575814.596853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575814.660846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575815.620913:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575815.684836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575816.644850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575816.708847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575817.220834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575817.220842:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575817.220846:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575817.220854:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7567 00000100:00080000:2.0:1788575817.220862:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575817.220868:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575817.220915:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864360704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575817.220959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575817.220963:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575817.220965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575817.220969:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7567 00000100:00080000:2.0:1788575817.220972:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575817.220976:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575817.220986:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575817.221000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864360832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575817.221018:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575817.221024:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864360704:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575817.221024:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575817.221027:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575817.221035:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7567 00000100:00080000:2.0:1788575817.221040:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575817.221043:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575817.221049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575817.221061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864360960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575817.221070:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575817.221074:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575817.221075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575817.221079:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7567 00000100:00080000:2.0:1788575817.221083:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575817.221085:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575817.221089:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575817.221099:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864361088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575817.221107:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575817.221108:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575817.221113:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575817.221115:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575817.221119:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7567 00000100:00080000:2.0:1788575817.221124:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575817.221128:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575817.221133:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575817.221145:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864361216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575817.221156:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff92fc4617f100 x1875449864360704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575817.221177:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864360704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575817.221185:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575817.221188:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575817.221190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864360704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788575817.221209:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575817.221226:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575817.221237:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff92fc4617f800 x1875449864360832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575817.221249:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864360832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575817.221258:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575817.221261:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575817.221263:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575817.221270:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788575817.221290:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575817.221301:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff92fc4617d180 x1875449864360960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575817.221312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864360960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575817.221320:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575817.221323:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575817.221327:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864360960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575817.221333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575817.221353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575817.221364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff92fc4617c380 x1875449864361088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575817.221375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864361088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575817.221388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575817.221391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575817.221394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788575817.221410:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788575817.221429:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575817.221440:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575817/real 1788575817] req@ffff92fc4617e680 x1875449864361216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575817.221452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864361216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575817.221467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575817.221470:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788575817.221473:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575817.668894:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575817.732846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575818.692917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575818.756824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575819.716893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575819.780824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575820.740873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575820.804862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575821.764978:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575821.828846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575822.340862:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575822.340874:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575822.340878:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575822.340889:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7572 00000100:00080000:2.0:1788575822.340898:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575822.340906:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575822.340948:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864361344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575822.340978:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575822.340984:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575822.340986:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575822.340991:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7572 00000100:00080000:2.0:1788575822.340995:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575822.340999:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575822.341009:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575822.341023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864361472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575822.341031:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864361344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575822.341038:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575822.341042:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575822.341045:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575822.341050:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7572 00000100:00080000:2.0:1788575822.341054:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575822.341059:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575822.341064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575822.341078:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864361600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575822.341088:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575822.341092:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788575822.341097:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575822.341103:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575822.341108:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7572 00000100:00080000:2.0:1788575822.341113:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575822.341118:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575822.341124:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575822.341137:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff92fc4617fb80 x1875449864361344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575822.341147:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864361728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575822.341154:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864361344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575822.341165:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575822.341170:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575822.341171:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575822.341172:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575822.341180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788575822.341181:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7572 00000100:00080000:2.0:1788575822.341188:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575822.341189:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864361344:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575822.341194:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575822.341201:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575822.341214:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864361856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575822.341218:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575822.341249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575822.341262:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff92fc4617c000 x1875449864361472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575822.341273:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864361472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575822.341287:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575822.341290:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575822.341293:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575822.341298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575822.341318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575822.341330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff92fc4617d880 x1875449864361600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575822.341340:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864361600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575822.341346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575822.341349:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575822.341352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575822.341358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575822.341375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575822.341387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff92fc4617ea00 x1875449864361728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575822.341397:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864361728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575822.341405:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575822.341418:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575822.341428:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575822.341446:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575822.341460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575822.341467:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575822/real 1788575822] req@ffff92fc4617e300 x1875449864361856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575822.341474:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864361856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575822.341479:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575822.341480:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575822.341482:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864361856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575822.788877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575822.852810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575823.812906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575823.876856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575824.836823:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575824.900810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575825.860902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575825.924838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575826.884859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575826.948844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575827.460861:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.460872:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575827.460876:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.460886:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7578 00000100:00080000:2.0:1788575827.460896:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.460903:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575827.460951:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864361984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575827.460981:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.460985:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575827.460987:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.460992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7578 00000100:00080000:2.0:1788575827.460996:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.461001:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575827.461011:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575827.461023:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864361984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575827.461028:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864362112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575827.461042:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.461045:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575827.461047:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.461051:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7578 00000100:00080000:2.0:1788575827.461055:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.461059:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575827.461064:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575827.461076:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864362240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575827.461086:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575827.461089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575827.461093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575827.461096:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.461101:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7578 00000100:00080000:2.0:1788575827.461105:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575827.461109:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575827.461113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575827.461126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864362368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575827.461132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff92fc4617ce00 x1875449864361984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575827.461161:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575827.461162:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864361984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575827.461169:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575827.461173:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575827.461181:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7578 00000100:00080000:1.0:1788575827.461183:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575827.461185:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575827.461190:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788575827.461193:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575827.461197:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864361984:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788575827.461201:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575827.461215:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864362496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575827.461219:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575827.461254:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575827.461268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff92fc4617d500 x1875449864362112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575827.461281:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864362112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575827.461290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575827.461293:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575827.461296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575827.461301:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575827.461315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575827.461322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff92fc4617e300 x1875449864362240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575827.461332:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864362240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575827.461339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575827.461342:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575827.461345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575827.461351:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575827.461369:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575827.461380:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff92fc4617ea00 x1875449864362368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575827.461394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864362368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575827.461399:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575827.461401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575827.461403:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575827.461416:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575827.461435:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575827.461458:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575827/real 1788575827] req@ffff92fc4617d880 x1875449864362496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575827.461470:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864362496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575827.461476:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575827.461478:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575827.461480:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575827.908945:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575827.974767:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575828.932858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575828.996828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575829.956918:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575830.020838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575830.980871:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575831.044823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575832.004905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575832.068863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575832.580926:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575832.580941:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575832.580947:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575832.580960:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7583 00000100:00080000:3.0:1788575832.580974:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575832.580984:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575832.581036:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864362624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575832.581072:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575832.581082:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575832.581084:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575832.581088:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7583 00000100:00080000:3.0:1788575832.581092:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575832.581096:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575832.581108:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575832.581124:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864362752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575832.581133:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864362624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575832.581149:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575832.581153:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575832.581155:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575832.581160:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7583 00000100:00080000:3.0:1788575832.581163:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575832.581167:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575832.581173:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575832.581185:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864362880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575832.581194:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575832.581197:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575832.581199:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575832.581203:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7583 00000100:00080000:3.0:1788575832.581206:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575832.581210:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575832.581215:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575832.581227:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864363008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575832.581232:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575832.581242:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575832.581246:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575832.581247:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575832.581252:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7583 00000100:00080000:3.0:1788575832.581257:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575832.581262:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575832.581267:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575832.581279:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864363136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575832.581288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff92fc67339f80 x1875449864362624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575832.581317:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864362624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575832.581328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575832.581332:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575832.581336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864362624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575832.581363:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575832.581381:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575832.581393:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff92fc6733ad80 x1875449864362752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575832.581406:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864362752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575832.581414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575832.581417:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575832.581421:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575832.581427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575832.581445:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575832.581454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff92fc67339180 x1875449864362880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575832.581465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864362880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575832.581479:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575832.581482:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575832.581485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864362880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575832.581490:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575832.581520:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575832.581529:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff92fc67339c00 x1875449864363008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575832.581539:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864363008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575832.581548:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575832.581551:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575832.581554:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575832.581559:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575832.581573:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575832.581581:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff92fc67339880 x1875449864363136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575832.581592:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864363136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575832.581598:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575832.581601:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575832.581605:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575833.028869:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575833.092836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575834.052847:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575834.116840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575835.076899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575835.140824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575836.100929:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575836.164845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575837.124905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575837.188833:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575837.700849:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575837.700859:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575837.700863:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575837.700873:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7588 00000100:00080000:3.0:1788575837.700884:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575837.700891:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575837.700925:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864363264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575837.700947:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575837.700950:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575837.700951:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575837.700953:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7588 00000100:00080000:3.0:1788575837.700955:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575837.700957:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575837.700964:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575837.700975:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864363392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575837.700984:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575837.700990:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575837.700991:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575837.700997:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7588 00000100:00100000:1.0:1788575837.700998:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864363264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575837.701002:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575837.701009:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575837.701013:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575837.701025:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864363520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575837.701033:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575837.701035:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575837.701036:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575837.701039:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7588 00000100:00080000:3.0:1788575837.701042:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575837.701044:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575837.701047:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575837.701055:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864363648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575837.701061:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575837.701063:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575837.701067:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575837.701068:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575837.701070:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7588 00000100:00080000:3.0:1788575837.701073:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575837.701075:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575837.701079:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575837.701085:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864363776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575837.701105:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff92fc6733b800 x1875449864363264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575837.701132:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864363264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575837.701143:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575837.701146:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575837.701150:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864363264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575837.701157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575837.701174:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575837.701184:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff92fc67338a80 x1875449864363392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575837.701205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864363392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575837.701213:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575837.701216:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575837.701219:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575837.701240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575837.701255:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575837.701263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff92fc67338380 x1875449864363520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575837.701274:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864363520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575837.701281:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575837.701284:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575837.701287:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575837.701291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575837.701304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575837.701311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff92fc67338000 x1875449864363648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575837.701321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864363648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575837.701328:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575837.701331:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575837.701334:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575837.701338:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575837.701350:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575837.701382:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff92fc6733aa00 x1875449864363776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575837.701401:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864363776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575837.701408:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575837.701411:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575837.701414:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864363776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575838.149207:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575838.212872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575839.172930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575839.236868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575840.196951:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575840.260864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575841.220921:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575841.285453:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575842.244948:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575842.308859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575842.820839:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575842.820850:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575842.820854:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575842.820864:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7593 00000100:00080000:3.0:1788575842.820873:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575842.820880:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575842.820914:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864363904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575842.820943:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575842.820946:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575842.820947:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575842.820951:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7593 00000100:00080000:3.0:1788575842.820955:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575842.820958:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575842.820991:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864363904:192.168.202.132@tcp:250:kworker.0 00000008:00100000:3.0:1788575842.821016:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575842.821031:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864364032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575842.821039:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575842.821041:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575842.821043:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575842.821045:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7593 00000100:00080000:3.0:1788575842.821049:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575842.821051:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575842.821054:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575842.821059:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575842.821067:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864364160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575842.821073:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575842.821075:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575842.821076:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575842.821078:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7593 00000100:00080000:3.0:1788575842.821081:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575842.821083:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575842.821086:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575842.821090:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff92fc67339500 x1875449864363904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575842.821096:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864364288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575842.821106:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575842.821108:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575842.821111:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575842.821112:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864363904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575842.821116:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7593 00000100:00080000:3.0:1788575842.821120:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575842.821126:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575842.821131:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575842.821136:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575842.821139:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864364416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575842.821145:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575842.821154:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864363904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575842.821182:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575842.821206:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575842.821236:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff92fc6733a300 x1875449864364032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.821255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864364032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.821263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.821266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575842.821270:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575842.821276:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575842.821295:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575842.821306:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff92fc6733b480 x1875449864364160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.821316:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864364160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.821323:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.821326:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575842.821329:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575842.821335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575842.821355:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575842.821364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff92fc67338700 x1875449864364288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.821375:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864364288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.821388:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.821391:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575842.821394:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575842.821400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575842.821419:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575842.821429:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff92fc6733bb80 x1875449864364416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575842.821440:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864364416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575842.821447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575842.821449:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575842.821451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575843.268852:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575843.332831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575844.292981:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575844.356886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575845.316889:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575845.380879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575846.340910:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575846.404834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575847.364847:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575847.428887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575847.940836:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.940846:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575847.940850:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.940859:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7598 00000100:00080000:3.0:1788575847.940868:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.940875:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575847.940908:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864364544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575847.940936:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.940940:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575847.940941:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.940944:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7598 00000100:00080000:3.0:1788575847.940948:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.940952:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575847.940960:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575847.940971:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864364672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575847.940982:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.940985:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575847.940987:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575847.940993:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864364544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788575847.940994:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7598 00000100:00080000:3.0:1788575847.940999:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.941005:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575847.941010:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575847.941021:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864364800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575847.941030:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.941033:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575847.941035:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.941039:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7598 00000100:00080000:3.0:1788575847.941042:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.941045:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575847.941049:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575847.941059:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864364928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575847.941067:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575847.941071:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788575847.941072:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575847.941076:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.941080:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7598 00000100:00080000:3.0:1788575847.941084:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.941089:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575847.941107:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575847.941119:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864365056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575847.941146:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff92fc6733b100 x1875449864364544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.941165:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864364544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.941175:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.941179:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575847.941183:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864364544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575847.941205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575847.941228:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575847.941240:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff92fc6733a680 x1875449864364672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.941251:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864364672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.941259:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.941262:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575847.941266:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575847.941284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575847.941304:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575847.941313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff92fc67338e00 x1875449864364800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.941324:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864364800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.941331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.941334:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575847.941337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575847.941342:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575847.941357:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575847.941364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff92fc6733bb80 x1875449864364928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.941376:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864364928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.941382:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.941385:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575847.941388:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864364928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575847.941392:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575847.941416:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575847.941426:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff92fc67338700 x1875449864365056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575847.941436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864365056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575847.941443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575847.941446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575847.941449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575848.390100:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575848.453563:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575849.412938:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575849.476863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575850.437027:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575850.500856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575851.460822:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575851.524834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575852.484920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575852.548854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575853.061068:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575853.061083:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575853.061089:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575853.061101:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7603 00000100:00080000:2.0:1788575853.061116:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575853.061125:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575853.061176:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864365184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575853.061209:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575853.061213:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575853.061215:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575853.061220:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7603 00000100:00080000:2.0:1788575853.061224:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575853.061231:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575853.061245:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575853.061259:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864365184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575853.061262:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864365312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575853.061284:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575853.061288:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575853.061291:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575853.061297:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7603 00000800:00000100:1.0:1788575853.061338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575853.061387:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575853/real 1788575853] req@ffff92fc4617c000 x1875449864365184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575853.061402:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864365184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575853.061409:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575853.061413:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575853.061416:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864365184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575853.061434:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575853.061460:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575853.061472:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575853/real 1788575853] req@ffff92fc4617fb80 x1875449864365312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575853.061484:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864365312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575853.061493:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575853.061496:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575853.061500:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365312:192.168.202.132@tcp:38:kworker.0 00000100:00080000:2.0:1788575853.061716:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575853.061759:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575853.061766:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575853.061783:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864365440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575853.061808:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575853.061812:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575853.061814:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575853.061818:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7603 00000100:00080000:2.0:1788575853.061823:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575853.061828:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575853.061834:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575853.061845:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575853.061847:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864365568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575853.061860:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575853.061864:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575853.061867:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575853.061871:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7603 00000100:00080000:2.0:1788575853.061874:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575853.061879:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575853.061883:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575853.061897:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788575853.061899:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864365696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575853.061998:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575853/real 1788575853] req@ffff92fc4617e680 x1875449864365440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575853.062014:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864365440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575853.062030:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575853.062035:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575853.062038:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575853.062063:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575853.062093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575853.062102:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575853/real 1788575853] req@ffff92fc4617c380 x1875449864365568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575853.062110:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864365568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575853.062115:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575853.062117:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575853.062119:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575853.062123:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575853.062141:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575853.062151:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575853/real 1788575853] req@ffff92fc4617d180 x1875449864365696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575853.062161:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864365696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575853.062168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575853.062171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575853.062175:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575853.509270:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575853.572857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575854.532882:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575854.596851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575855.556850:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575855.620834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575856.581975:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575856.644891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575857.604920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575857.668928:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575858.180851:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.180865:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575858.180871:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.180898:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7608 00000100:00080000:2.0:1788575858.180912:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.180922:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575858.180979:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864365824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.181019:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.181024:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575858.181026:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.181030:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7608 00000100:00080000:2.0:1788575858.181034:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.181038:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575858.181050:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575858.181066:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864365952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.181089:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.181093:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575858.181095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.181099:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7608 00000100:00080000:2.0:1788575858.181103:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.181107:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575858.181113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575858.181126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864366080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.181136:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.181139:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575858.181142:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.181145:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7608 00000100:00080000:2.0:1788575858.181149:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.181153:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575858.181158:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575858.181181:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864366208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575858.181190:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575858.181193:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575858.181195:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575858.181199:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7608 00000100:00080000:2.0:1788575858.181204:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575858.181209:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575858.181214:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575858.181226:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864366336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575858.182138:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864365824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575858.182283:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575858.182313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575858.182340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575858.182360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575858.628957:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575858.692891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575859.652919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575859.717012:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575860.676968:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575860.740898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788575861.253054:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788575861.253111:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575861.253129:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575861.253177:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575861.253185:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575861.253193:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788575861.253199:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788575861.253406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff92fc4617f800 x1875449864365824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575861.253437:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864365824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575861.253449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575861.253454:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575861.253457:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864365824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575861.253464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff92fc4617f100 x1875449864365952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575861.253476:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864365952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575861.253483:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575861.253486:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575861.253490:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864365952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575861.253496:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff92fc4617dc00 x1875449864366080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575861.253509:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864366080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575861.253518:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575861.253522:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575861.253525:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575861.253531:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff92fc4617ed80 x1875449864366208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575861.253543:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864366208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575861.253552:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575861.253556:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575861.253559:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575861.253567:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575858/real 1788575861] req@ffff92fc4617c700 x1875449864366336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575861.253578:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864366336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575861.253588:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575861.253592:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575861.253603:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575861.701332:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575861.764883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575862.724930:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575862.788891:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575863.300950:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.300961:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575863.300965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.300976:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7613 00000100:00080000:2.0:1788575863.300987:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.300995:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575863.301032:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864366464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575863.301066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.301071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575863.301072:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.301076:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7613 00000100:00080000:2.0:1788575863.301080:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.301084:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575863.301093:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575863.301106:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864366592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575863.301115:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.301121:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575863.301122:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.301125:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7613 00000100:00080000:2.0:1788575863.301129:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.301132:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575863.301137:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575863.301149:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864366720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575863.301158:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.301161:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575863.301162:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.301165:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7613 00000100:00080000:2.0:1788575863.301169:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.301172:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575863.301176:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575863.301186:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864366848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575863.301194:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575863.301197:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575863.301199:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575863.301202:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7613 00000100:00080000:2.0:1788575863.301204:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575863.301207:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575863.301212:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575863.301221:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864366976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575863.304109:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864366464:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575863.304170:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575863.304206:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff92fc4617ca80 x1875449864366464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575863.304226:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864366464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575863.304236:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575863.304241:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575863.304244:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864366464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575863.304252:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575863.304282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575863.304295:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff92fc4617f480 x1875449864366592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575863.304307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864366592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575863.304315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575863.304318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575863.304321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575863.304328:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575863.304346:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575863.304357:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff92fc4617c700 x1875449864366720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575863.304368:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864366720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575863.304381:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575863.304384:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575863.304387:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575863.304394:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575863.304412:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575863.304423:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff92fc4617ed80 x1875449864366848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575863.304433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864366848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575863.304441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575863.304444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575863.304447:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575863.304453:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575863.304471:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575863.304480:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575863/real 1788575863] req@ffff92fc4617dc00 x1875449864366976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575863.304490:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864366976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575863.304497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575863.304500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575863.304504:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864366976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575863.748924:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575863.812858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575864.774251:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575864.837037:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575865.797125:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575865.860834:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575866.820868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575866.884810:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575867.844846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575867.908837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575868.420840:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.420850:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575868.420854:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.420863:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7619 00000100:00080000:1.0:1788575868.420873:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575868.420880:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575868.420912:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864367104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575868.420937:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.420941:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575868.420942:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.420946:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7619 00000100:00080000:1.0:1788575868.420949:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575868.420952:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575868.420961:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575868.420972:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864367232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575868.420983:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575868.420987:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864367104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575868.420988:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575868.420991:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.420998:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7619 00000100:00080000:1.0:1788575868.421001:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575868.421004:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575868.421009:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575868.421020:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864367360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575868.421028:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.421031:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575868.421033:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.421037:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7619 00000100:00080000:1.0:1788575868.421041:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575868.421048:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788575868.421048:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575868.421055:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575868.421067:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864367488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575868.421076:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575868.421079:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575868.421081:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575868.421085:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7619 00000100:00080000:1.0:1788575868.421089:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575868.421093:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff92fc6728b100 x1875449864367104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575868.421095:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575868.421103:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575868.421117:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864367616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575868.421121:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864367104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575868.421141:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575868.421146:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575868.421151:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864367104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575868.421174:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575868.421194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575868.421204:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff92fc67289880 x1875449864367232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575868.421214:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864367232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575868.421222:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575868.421224:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575868.421227:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575868.421233:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575868.421246:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575868.421255:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff92fc6728ad80 x1875449864367360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575868.421266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864367360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575868.421279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575868.421282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575868.421285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575868.421291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575868.421309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575868.421319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff92fc6728b800 x1875449864367488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575868.421329:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864367488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575868.421336:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575868.421339:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575868.421342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575868.421347:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575868.421364:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575868.421375:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575868/real 1788575868] req@ffff92fc67289f80 x1875449864367616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575868.421385:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864367616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575868.421393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575868.421396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575868.421399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575868.868885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575868.933854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575869.892868:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575869.956849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575870.916890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575870.980843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575871.940877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575872.004840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575872.965389:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575873.028836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575873.540828:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575873.540836:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575873.540839:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575873.540847:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7624 00000100:00080000:1.0:1788575873.540855:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575873.540860:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575873.540904:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864367744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575873.540934:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575873.540938:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575873.540939:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575873.540945:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7624 00000100:00080000:1.0:1788575873.540949:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575873.540952:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575873.540962:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575873.540977:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864367872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575873.540988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864367744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575873.540989:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575873.540994:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575873.540996:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575873.541001:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7624 00000100:00080000:1.0:1788575873.541005:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575873.541009:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575873.541013:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575873.541027:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864368000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575873.541036:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575873.541039:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575873.541041:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575873.541044:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7624 00000800:00000100:3.0:1788575873.541049:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575873.541050:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575873.541064:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575873.541072:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575873.541084:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864368128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575873.541091:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575873.541095:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575873.541095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff92fc6728a680 x1875449864367744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575873.541098:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575873.541102:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7624 00000100:00080000:1.0:1788575873.541105:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575873.541108:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575873.541114:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575873.541125:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864368256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575873.541134:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864367744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575873.541146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575873.541150:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575873.541155:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864367744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575873.541177:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575873.541200:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575873.541212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff92fc67289c00 x1875449864367872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575873.541224:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864367872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575873.541232:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575873.541235:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575873.541238:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864367872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575873.541254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575873.541274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575873.541293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff92fc67288380 x1875449864368000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575873.541306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864368000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575873.541312:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575873.541315:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575873.541323:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575873.541329:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575873.541349:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575873.541361:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff92fc6728bb80 x1875449864368128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575873.541372:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864368128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575873.541378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575873.541380:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575873.541382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575873.541386:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575873.541399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575873.541412:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575873/real 1788575873] req@ffff92fc67288700 x1875449864368256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575873.541420:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864368256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575873.541424:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575873.541426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575873.541428:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575873.988899:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575874.052818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575875.012919:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575875.076824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575876.036896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575876.101785:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575877.060905:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575877.124812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575878.084956:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575878.148866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575878.660890:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.660901:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575878.660906:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.660917:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7629 00000100:00080000:1.0:1788575878.660929:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.660936:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575878.660977:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864368384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.661006:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.661009:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.661010:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.661013:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7629 00000100:00080000:1.0:1788575878.661016:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.661019:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575878.661029:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575878.661038:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864368512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.661045:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.661047:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575878.661048:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.661051:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7629 00000100:00080000:1.0:1788575878.661053:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.661055:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575878.661058:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575878.661064:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864368384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575878.661066:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864368640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.661076:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.661078:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.661079:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.661081:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7629 00000100:00080000:1.0:1788575878.661083:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.661085:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575878.661087:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575878.661097:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864368768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.661107:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.661110:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575878.661111:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.661113:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7629 00000100:00080000:1.0:1788575878.661116:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.661118:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575878.661121:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575878.661126:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864368896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575878.661202:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575878.661258:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff92fc67288a80 x1875449864368384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575878.661279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864368384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575878.661290:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575878.661294:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575878.661299:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864368384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575878.661307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575878.661330:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575878.661343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff92fc67289500 x1875449864368512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575878.661354:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864368512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575878.661370:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575878.661374:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575878.661377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575878.661406:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575878.661428:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575878.661440:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff92fc67289180 x1875449864368640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575878.661451:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864368640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575878.661460:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575878.661491:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575878.661494:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575878.661501:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575878.661520:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575878.661529:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff92fc67288700 x1875449864368768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575878.661540:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864368768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575878.661548:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575878.661551:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575878.661555:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575878.661560:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575878.661576:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575878.661585:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff92fc6728bb80 x1875449864368896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575878.661595:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864368896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575878.661603:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575878.661605:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575878.661609:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864368896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575879.108989:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575879.172822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575880.134761:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575880.196823:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575881.156944:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575881.220905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575882.180920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575882.245056:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575883.204935:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575883.268923:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575883.780834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.780845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575883.780849:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.780859:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7634 00000100:00080000:0.0:1788575883.780869:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.780877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575883.780918:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864369024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.780950:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.780954:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575883.780956:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.780960:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7634 00000100:00080000:0.0:1788575883.780964:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.780968:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575883.780978:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575883.780991:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864369152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.780999:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.781001:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575883.781002:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.781005:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7634 00000100:00080000:0.0:1788575883.781007:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.781010:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575883.781013:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575883.781022:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864369280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.781030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.781033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575883.781035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.781038:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7634 00000100:00080000:0.0:1788575883.781041:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.781045:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575883.781049:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575883.781058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864369408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.781063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.781065:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575883.781067:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.781068:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7634 00000100:00080000:0.0:1788575883.781070:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.781072:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575883.781075:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575883.781080:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864369536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575883.784166:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864369024:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575883.784231:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575883.784271:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff92fc6704b100 x1875449864369024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575883.784285:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864369024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575883.784293:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575883.784296:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575883.784300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864369024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575883.784305:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575883.784320:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575883.784328:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff92fc67049500 x1875449864369152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575883.784336:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864369152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575883.784341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575883.784343:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575883.784345:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575883.784348:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575883.784376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575883.784383:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff92fc6704bb80 x1875449864369280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575883.784390:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864369280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575883.784395:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575883.784397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575883.784399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575883.784402:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575883.784413:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575883.784420:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff92fc67049180 x1875449864369408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575883.784431:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864369408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575883.784439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575883.784442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575883.784445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575883.784450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575883.784470:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575883.784480:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff92fc67048e00 x1875449864369536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575883.784491:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864369536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575883.784499:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575883.784502:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575883.784505:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575884.228841:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575884.292851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575885.253298:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575885.316830:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575886.276822:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575886.340859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575887.300899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575887.364836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575888.324860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575888.388842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575888.900865:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.900876:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575888.900881:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.900890:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7639 00000100:00080000:1.0:1788575888.900900:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.900907:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575888.900953:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864369664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575888.900987:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.900991:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575888.900993:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.900997:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7639 00000100:00080000:1.0:1788575888.901001:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.901005:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575888.901014:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575888.901030:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864369792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575888.901040:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.901043:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575888.901045:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.901048:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7639 00000100:00080000:1.0:1788575888.901051:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.901055:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575888.901060:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575888.901070:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864369920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575888.901087:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.901090:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575888.901092:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575888.901096:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864369664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575888.901099:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7639 00000100:00080000:1.0:1788575888.901107:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.901111:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575888.901116:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575888.901129:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864370048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575888.901137:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575888.901141:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575888.901143:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575888.901146:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7639 00000100:00080000:1.0:1788575888.901157:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.901162:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788575888.901166:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575888.901170:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575888.901184:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864370176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575888.901212:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff92fc67288380 x1875449864369664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575888.901230:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864369664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575888.901240:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575888.901244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575888.901248:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864369664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575888.901274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575888.901298:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575888.901311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff92fc67289c00 x1875449864369792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575888.901322:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864369792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575888.901330:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575888.901333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575888.901336:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575888.901359:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575888.901378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575888.901389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff92fc6728a680 x1875449864369920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575888.901400:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864369920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575888.901407:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575888.901410:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575888.901413:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864369920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575888.901418:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575888.901433:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575888.901442:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff92fc67289f80 x1875449864370048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575888.901452:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864370048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575888.901459:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575888.901462:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575888.901464:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575888.901469:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575888.901483:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575888.901492:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff92fc6728b800 x1875449864370176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575888.901500:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864370176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575888.901507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575888.901510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575888.901513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575889.348927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575889.412847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575890.372862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575890.436837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575891.397055:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575891.460824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575892.420864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575892.484839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575893.444902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575893.508836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575894.020816:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575894.020823:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575894.020826:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575894.020834:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7644 00000100:00080000:1.0:1788575894.020843:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575894.020849:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575894.020880:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864370304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575894.020906:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575894.020910:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575894.020911:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575894.020915:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7644 00000100:00080000:1.0:1788575894.020918:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575894.020921:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575894.020929:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575894.020940:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864370432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575894.020950:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575894.020953:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575894.020955:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575894.020959:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7644 00000100:00080000:1.0:1788575894.020963:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575894.020968:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575894.020975:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575894.020987:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864370560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575894.020996:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575894.021000:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575894.021003:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575894.021006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864370304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575894.021008:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7644 00000100:00080000:1.0:1788575894.021016:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575894.021028:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575894.021032:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575894.021045:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864370688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575894.021052:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575894.021054:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575894.021056:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575894.021060:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7644 00000100:00080000:1.0:1788575894.021066:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575894.021070:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575894.021075:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575894.021091:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864370816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575894.021358:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575894.021470:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575894/real 1788575894] req@ffff92fc6728ad80 x1875449864370304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575894.021491:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864370304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575894.021509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575894.021513:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575894.021517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864370304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575894.021525:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575894.021549:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575894.021563:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575894/real 1788575894] req@ffff92fc67289880 x1875449864370432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575894.021583:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864370432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575894.021591:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575894.021594:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575894.021598:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575894.025295:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575894.025374:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575894.025404:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575894/real 1788575894] req@ffff92fc6728b100 x1875449864370560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575894.025424:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864370560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575894.025435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575894.025440:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575894.025444:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575894.025452:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575894.025473:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575894.025485:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575894/real 1788575894] req@ffff92fc67288000 x1875449864370688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575894.025496:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864370688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575894.025504:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575894.025507:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575894.025511:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575894.025518:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575894.025537:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575894.025547:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575894/real 1788575894] req@ffff92fc67288e00 x1875449864370816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575894.025559:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864370816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575894.025567:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575894.025570:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575894.025573:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864370816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575894.468929:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575894.532848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575895.493274:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575895.556872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575896.517244:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575896.580889:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575897.540943:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575897.604910:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575898.565880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575898.628905:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575899.141143:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575899.141159:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575899.141165:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575899.141177:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7649 00000100:00080000:1.0:1788575899.141189:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575899.141197:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575899.141248:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864370944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575899.141286:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575899.141290:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575899.141292:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575899.141296:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7649 00000100:00080000:1.0:1788575899.141300:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575899.141304:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575899.141315:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575899.141327:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864371072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575899.141337:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575899.141340:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575899.141342:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575899.141346:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7649 00000100:00080000:1.0:1788575899.141349:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575899.141352:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575899.141357:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575899.141367:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864371200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575899.141376:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575899.141380:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575899.141382:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575899.141385:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7649 00000100:00080000:1.0:1788575899.141388:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575899.141392:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575899.141397:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575899.141408:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864371328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575899.141417:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575899.141420:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575899.141421:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575899.141424:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7649 00000100:00080000:1.0:1788575899.141427:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575899.141430:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575899.141435:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575899.141444:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864371456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575899.143135:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864370944:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575899.143219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575899.143274:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575899/real 1788575899] req@ffff92fc6728aa00 x1875449864370944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575899.143296:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864370944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575899.143307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575899.143312:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575899.143315:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864370944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575899.143323:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575899.143345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575899.143358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575899/real 1788575899] req@ffff92fc6728b480 x1875449864371072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575899.143370:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864371072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575899.143378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575899.143382:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575899.143385:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575899.143391:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575899.143411:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575899.143421:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575899/real 1788575899] req@ffff92fc6728a300 x1875449864371200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575899.143433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864371200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575899.143447:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575899.143450:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575899.143453:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575899.143459:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575899.143480:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575899.143490:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575899/real 1788575899] req@ffff92fc67288e00 x1875449864371328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575899.143501:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864371328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575899.143510:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575899.143512:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575899.143515:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575899.143521:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575899.143540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575899.143552:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575899/real 1788575899] req@ffff92fc67288000 x1875449864371456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575899.143563:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864371456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575899.143571:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575899.143574:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575899.143591:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575899.589244:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575899.652883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575900.612986:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575900.676900:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575901.636922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575901.700907:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575902.660907:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575902.725310:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575903.685816:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575903.748857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575904.261284:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575904.261296:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575904.261301:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575904.261315:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7654 00000100:00080000:0.0:1788575904.261329:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575904.261338:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575904.261385:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864371584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575904.261427:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575904.261431:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575904.261433:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575904.261437:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7654 00000100:00080000:0.0:1788575904.261441:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575904.261445:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575904.261458:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575904.261470:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864371712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575904.261479:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575904.261482:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575904.261484:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575904.261487:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7654 00000100:00080000:0.0:1788575904.261490:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575904.261494:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575904.261498:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575904.261507:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864371840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575904.261516:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575904.261519:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575904.261520:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575904.261524:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7654 00000100:00080000:0.0:1788575904.261528:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575904.261531:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575904.261535:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575904.261544:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864371968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575904.261552:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575904.261555:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575904.261556:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575904.261559:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7654 00000100:00080000:0.0:1788575904.261563:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575904.261566:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575904.261571:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575904.261580:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864372096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575904.262149:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864371584:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575904.262224:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575904.262270:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff92fc67049c00 x1875449864371584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575904.262289:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864371584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575904.262300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575904.262304:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575904.262308:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864371584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575904.262315:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575904.262333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575904.262344:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff92fc67049880 x1875449864371712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575904.262355:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864371712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575904.262364:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575904.262366:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575904.262370:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575904.262375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575904.262390:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575904.262400:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff92fc6704ad80 x1875449864371840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575904.262411:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864371840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575904.262419:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575904.262422:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575904.262424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575904.262430:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575904.262446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575904.262454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff92fc6704a680 x1875449864371968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575904.262465:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864371968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575904.262472:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575904.262476:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575904.262478:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864371968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575904.262483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575904.262497:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575904.262505:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575904/real 1788575904] req@ffff92fc6704aa00 x1875449864372096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575904.262515:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864372096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575904.262524:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575904.262527:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575904.262530:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575904.708874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575904.772882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575905.733713:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575905.796982:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575906.756862:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575906.820926:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575907.781311:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575907.844897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575908.804984:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575908.868945:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575909.380919:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575909.380930:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575909.380935:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575909.380945:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7660 00000100:00080000:1.0:1788575909.380955:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575909.380974:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575909.381012:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864372224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575909.381044:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575909.381048:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575909.381050:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575909.381054:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7660 00000100:00080000:1.0:1788575909.381058:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575909.381062:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575909.381072:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575909.381085:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864372352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575909.381095:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575909.381098:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575909.381100:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575909.381105:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7660 00000100:00080000:1.0:1788575909.381108:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575909.381112:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575909.381117:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575909.381127:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864372480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575909.381134:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575909.381137:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575909.381138:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575909.381142:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7660 00000100:00080000:1.0:1788575909.381144:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575909.381147:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575909.381152:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575909.381159:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864372608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575909.381168:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575909.381171:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575909.381173:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575909.381176:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7660 00000100:00080000:1.0:1788575909.381179:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575909.381183:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575909.381188:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575909.381198:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864372736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575909.381340:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864372224:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575909.381399:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575909.381450:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff92fc6728b100 x1875449864372224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575909.381467:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864372224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575909.381477:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575909.381481:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575909.381485:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864372224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575909.381492:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575909.381515:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575909.381527:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff92fc67289880 x1875449864372352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575909.381536:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864372352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575909.381557:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575909.381561:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575909.381564:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575909.381570:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575909.381589:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575909.381599:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff92fc6728ad80 x1875449864372480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575909.381610:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864372480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575909.381618:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575909.381621:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575909.381624:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575909.381629:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575909.381648:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575909.381659:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff92fc6728b800 x1875449864372608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575909.381669:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864372608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575909.381675:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575909.381678:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575909.381681:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575909.381686:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575909.381702:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575909.381711:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575909/real 1788575909] req@ffff92fc67289f80 x1875449864372736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575909.381758:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864372736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575909.381766:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575909.381769:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575909.381771:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575909.828954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575909.892866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575910.852917:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575910.916902:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575911.876906:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575911.940852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575912.900882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575912.964888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575913.924993:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575913.988908:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575914.501080:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.501087:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575914.501090:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.501098:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7665 00000100:00080000:2.0:1788575914.501106:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.501112:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575914.501142:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864372864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.501170:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.501174:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575914.501175:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.501178:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7665 00000100:00080000:2.0:1788575914.501180:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.501182:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575914.501188:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575914.501196:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864372992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.501201:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.501203:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575914.501204:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.501206:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7665 00000100:00080000:2.0:1788575914.501208:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.501210:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575914.501212:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575914.501217:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864373120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.501224:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.501226:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575914.501228:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.501231:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7665 00000100:00080000:2.0:1788575914.501234:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.501237:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575914.501241:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575914.501250:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864373248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575914.501258:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575914.501261:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575914.501263:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575914.501266:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7665 00000100:00080000:2.0:1788575914.501269:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575914.501272:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575914.501275:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575914.501281:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864373376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575914.501360:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864372864:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575914.501405:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575914.501455:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff92fc4617f100 x1875449864372864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575914.501471:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864372864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575914.501480:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575914.501483:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575914.501486:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864372864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575914.501493:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575914.501513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575914.501523:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff92fc4617f800 x1875449864372992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575914.501532:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864372992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575914.501544:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575914.501547:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575914.501550:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864372992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575914.501555:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575914.501569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575914.501578:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff92fc4617d180 x1875449864373120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575914.501587:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864373120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575914.501593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575914.501596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575914.501599:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575914.501603:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575914.501619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575914.501627:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff92fc4617c380 x1875449864373248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575914.501637:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864373248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575914.501643:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575914.501646:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575914.501649:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575914.501653:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575914.501667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575914.501676:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575914/real 1788575914] req@ffff92fc4617e680 x1875449864373376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575914.501686:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864373376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575914.501693:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575914.501695:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575914.501698:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575914.948987:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575915.012883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575915.973257:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575916.036856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575916.996964:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575917.060882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575918.021534:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575918.084882:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575919.044911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575919.108924:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575919.620941:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.620951:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575919.620955:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.620971:0:40043:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7670 00000100:00080000:1.0:1788575919.620984:0:40043:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.620991:0:40043:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575919.621025:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864373504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.621061:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.621066:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575919.621067:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.621071:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7670 00000100:00080000:1.0:1788575919.621075:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.621078:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575919.621088:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575919.621100:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864373632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.621112:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575919.621116:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864373504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575919.621117:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575919.621119:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.621127:0:40043:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7670 00000100:00080000:1.0:1788575919.621131:0:40043:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.621134:0:40043:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575919.621138:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575919.621149:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864373760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.621165:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.621168:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575919.621170:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.621173:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7670 00000100:00080000:1.0:1788575919.621176:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.621180:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788575919.621184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575919.621186:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575919.621198:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864373888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.621207:0:40043:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.621210:0:40043:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575919.621212:0:40043:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.621215:0:40043:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7670 00000100:00080000:1.0:1788575919.621219:0:40043:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.621223:0:40043:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575919.621227:0:40043:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575919.621231:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff92fc6728a680 x1875449864373504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575919.621241:0:40043:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864374016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575919.621255:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864373504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575919.621264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575919.621267:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575919.621269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864373504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575919.621289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575919.621305:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575919.621313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff92fc67289c00 x1875449864373632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575919.621321:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864373632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575919.621326:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575919.621328:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575919.621330:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575919.621333:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575919.621343:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575919.621349:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff92fc67288380 x1875449864373760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575919.621356:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864373760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575919.621360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575919.621361:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575919.621363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575919.621366:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575919.621375:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575919.621380:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff92fc6728bb80 x1875449864373888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575919.621387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864373888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575919.621392:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575919.621393:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575919.621395:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864373888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575919.621398:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575919.621407:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575919.621411:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff92fc67288700 x1875449864374016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575919.621417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864374016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575919.621421:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575919.621423:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575919.621425:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575920.068892:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575920.132848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575921.092858:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575921.156840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575922.116904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575922.180839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575923.140904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575923.204828:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575924.164954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575924.228858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575924.740834:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.740845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575924.740849:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.740860:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7675 00000100:00080000:0.0:1788575924.740871:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.740880:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575924.740929:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864374144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575924.740965:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.740969:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575924.740971:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.740975:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7675 00000100:00080000:0.0:1788575924.740978:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.740982:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575924.740994:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575924.741007:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864374272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575924.741017:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.741020:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575924.741022:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.741025:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7675 00000100:00080000:0.0:1788575924.741029:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.741033:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575924.741038:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575924.741048:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864374400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575924.741063:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.741066:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575924.741068:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.741072:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7675 00000100:00080000:0.0:1788575924.741075:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.741079:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575924.741085:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575924.741095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864374528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575924.741101:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575924.741104:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575924.741105:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575924.741107:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7675 00000100:00080000:0.0:1788575924.741110:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575924.741113:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575924.741117:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575924.741126:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864374656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575924.741450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864374144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575924.741565:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575924.741598:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575924.741623:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575924.741642:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575925.188941:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575925.253001:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575926.212960:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575926.276826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575927.236931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575927.300872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788575927.813152:0:2383:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788575927.813218:0:2383:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575927.813238:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575927.813293:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575927.813304:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575927.813313:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788575927.813321:0:2383:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788575927.813468:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff92fc6704a300 x1875449864374144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575927.813516:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864374144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575927.813530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575927.813536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575927.813541:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864374144:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575927.813550:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff92fc67049f80 x1875449864374272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575927.813564:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864374272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575927.813572:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575927.813576:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575927.813580:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575927.813588:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff92fc67048700 x1875449864374400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575927.813601:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864374400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575927.813610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575927.813613:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575927.813617:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575927.813624:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff92fc67048380 x1875449864374528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575927.813637:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864374528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575927.813645:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575927.813648:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575927.813651:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575927.813669:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575927] req@ffff92fc67048000 x1875449864374656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575927.813683:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864374656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575927.813692:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575927.813696:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575927.813699:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575928.260940:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575928.324869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575929.284904:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575929.348771:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575929.860950:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575929.860959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575929.860975:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575929.860991:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7680 00000100:00080000:0.0:1788575929.861005:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575929.861015:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575929.861060:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864374784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575929.861095:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575929.861100:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575929.861102:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575929.861106:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7680 00000100:00080000:0.0:1788575929.861110:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575929.861115:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575929.861130:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575929.861150:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864374912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575929.861160:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575929.861164:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575929.861165:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575929.861171:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7680 00000100:00080000:0.0:1788575929.861175:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575929.861178:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575929.861184:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575929.861196:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864375040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575929.861211:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575929.861214:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575929.861222:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575929.861226:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7680 00000100:00080000:0.0:1788575929.861230:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575929.861234:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575929.861243:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575929.861256:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864375168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575929.861258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864374784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575929.861268:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575929.861273:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575929.861274:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575929.861279:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7680 00000100:00080000:0.0:1788575929.861282:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575929.861287:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575929.861293:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575929.861307:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864375296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575929.861387:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575929.863966:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff92fc67048a80 x1875449864374784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575929.864007:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864374784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.864020:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575929.864024:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575929.864030:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864374784:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575929.864043:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374912:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575929.864102:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575929.864126:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff92fc67048000 x1875449864374912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575929.864138:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864374912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.864146:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575929.864150:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575929.864152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864374912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575929.864157:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375040:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575929.864173:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575929.864181:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff92fc67048380 x1875449864375040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575929.864188:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864375040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.864193:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575929.864195:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575929.864197:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575929.864222:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375168:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575929.864235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575929.864241:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff92fc67048700 x1875449864375168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575929.864259:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864375168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.864264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575929.864266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575929.864268:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575929.864272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375296:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575929.864282:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575929.864288:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff92fc67049f80 x1875449864375296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575929.864295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864375296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.864300:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575929.864302:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575929.864304:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575930.308982:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575930.372824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575931.332913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575931.396866:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575932.358709:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575932.420870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575933.380927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575933.444876:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575934.404931:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575934.468873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575934.980862:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575934.980874:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575934.980879:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575934.980889:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7685 00000100:00080000:1.0:1788575934.980900:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575934.980907:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575934.980958:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864375424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575934.980983:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575934.980987:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575934.980989:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575934.980994:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7685 00000100:00080000:1.0:1788575934.980998:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575934.981001:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575934.981009:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575934.981022:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864375552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575934.981033:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575934.981035:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864375424:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575934.981037:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575934.981041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575934.981047:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7685 00000100:00080000:1.0:1788575934.981050:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575934.981053:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575934.981059:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575934.981071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864375680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575934.981079:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575934.981082:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575934.981084:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575934.981089:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7685 00000100:00080000:1.0:1788575934.981092:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575934.981098:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788575934.981099:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575934.981107:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575934.981121:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864375808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575934.981130:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575934.981133:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575934.981135:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575934.981140:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7685 00000100:00100000:3.0:1788575934.981141:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff92fc67289180 x1875449864375424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575934.981145:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575934.981153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575934.981161:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575934.981168:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864375424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575934.981176:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864375936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575934.981182:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575934.981189:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575934.981194:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864375424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575934.981216:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575934.981235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575934.981247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff92fc67289500 x1875449864375552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575934.981257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864375552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575934.981264:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575934.981266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575934.981269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375552:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575934.981274:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575934.981289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575934.981298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff92fc67288a80 x1875449864375680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575934.981307:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864375680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575934.981315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575934.981318:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575934.981321:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575934.981326:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575934.981342:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575934.981351:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff92fc67288700 x1875449864375808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575934.981361:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864375808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575934.981369:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575934.981372:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575934.981375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575934.981380:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575934.981396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575934.981405:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff92fc6728bb80 x1875449864375936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575934.981416:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864375936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575934.981423:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575934.981426:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575934.981429:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864375936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575935.428906:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575935.493083:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575936.452854:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575936.516873:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575937.476922:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575937.540874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575938.500927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575938.564880:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575939.524853:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575939.588892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575940.100895:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575940.100909:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575940.100914:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575940.100924:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7690 00000100:00080000:2.0:1788575940.100936:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575940.100944:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575940.100990:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864376064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575940.101024:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575940.101028:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575940.101029:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575940.101032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7690 00000100:00080000:2.0:1788575940.101035:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575940.101038:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575940.101048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575940.101058:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864376192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575940.101066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575940.101068:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575940.101070:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575940.101072:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7690 00000100:00080000:2.0:1788575940.101074:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575940.101076:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575940.101079:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575940.101085:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864376320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575940.101091:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575940.101094:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575940.101095:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575940.101097:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7690 00000100:00080000:2.0:1788575940.101099:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575940.101101:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575940.101103:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575940.101110:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864376448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575940.101115:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575940.101116:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575940.101117:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575940.101119:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7690 00000100:00080000:2.0:1788575940.101122:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575940.101124:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575940.101127:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575940.101132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864376576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575940.101307:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864376064:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575940.101383:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575940.101441:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575940/real 1788575940] req@ffff92fc4617fb80 x1875449864376064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575940.101459:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864376064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575940.101469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575940.101473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575940.101477:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864376064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575940.101483:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575940.101502:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575940.101513:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575940/real 1788575940] req@ffff92fc4617c000 x1875449864376192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575940.101522:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864376192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575940.101530:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575940.101533:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575940.101536:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575940.101541:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575940.101588:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575940.101598:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575940/real 1788575940] req@ffff92fc4617d880 x1875449864376320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575940.101608:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864376320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575940.101620:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575940.101623:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575940.101626:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575940.101631:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575940.101646:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575940.101654:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575940/real 1788575940] req@ffff92fc4617ea00 x1875449864376448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575940.101663:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864376448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575940.101671:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575940.101674:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575940.101677:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575940.101682:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575940.101698:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575940.101707:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575940/real 1788575940] req@ffff92fc4617e300 x1875449864376576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575940.101717:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864376576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575940.101744:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575940.101747:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575940.101750:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575940.548938:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575940.612864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575941.572903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575941.636860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575942.596954:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575942.660819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575943.620840:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575943.684888:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575944.644878:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575944.708797:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575945.220878:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575945.220888:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575945.220892:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575945.220903:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7695 00000100:00080000:2.0:1788575945.220913:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575945.220922:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575945.220958:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864376704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575945.220992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575945.220997:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575945.220999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575945.221002:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7695 00000100:00080000:2.0:1788575945.221006:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575945.221010:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575945.221019:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575945.221032:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864376832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575945.221042:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575945.221046:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575945.221048:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575945.221051:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7695 00000100:00080000:2.0:1788575945.221055:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575945.221058:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575945.221062:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575945.221291:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864376960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575945.221303:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575945.221306:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575945.221308:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575945.221312:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7695 00000100:00080000:2.0:1788575945.221316:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575945.221320:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575945.221325:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575945.221338:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864377088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575945.221348:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575945.221350:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575945.221352:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575945.221355:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7695 00000100:00080000:2.0:1788575945.221358:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575945.221361:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575945.221365:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575945.221375:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864377216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575945.221560:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864376704:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575945.221616:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575945.221656:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff92fc4617d500 x1875449864376704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575945.221674:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864376704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575945.221685:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575945.221689:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575945.221693:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864376704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575945.221700:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575945.221741:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575945.221755:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff92fc4617ce00 x1875449864376832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575945.221766:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864376832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575945.221773:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575945.221776:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575945.221779:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575945.221784:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575945.221802:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575945.221812:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff92fc4617e300 x1875449864376960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575945.221822:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864376960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575945.221829:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575945.221832:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575945.221835:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864376960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575945.221840:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575945.221857:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575945.221866:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff92fc4617ea00 x1875449864377088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575945.221876:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864377088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575945.221883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575945.221886:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575945.221889:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575945.221894:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575945.221911:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575945.221921:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575945/real 1788575945] req@ffff92fc4617d880 x1875449864377216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575945.221931:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864377216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575945.221938:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575945.221941:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575945.221944:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575945.668864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575945.732825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575946.692897:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575946.756844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575947.716825:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575947.780835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575948.740980:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575948.804872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575949.764834:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575949.828886:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575950.340842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.340854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575950.340858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.340868:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7700 00000100:00080000:2.0:1788575950.340879:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.340886:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575950.340921:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864377344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575950.340950:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.340954:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575950.340956:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.340960:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7700 00000100:00080000:2.0:1788575950.340963:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.340967:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575950.340976:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575950.340988:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864377472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575950.340999:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.341003:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575950.341004:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.341008:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7700 00000100:00080000:2.0:1788575950.341013:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.341016:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575950.341022:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575950.341040:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864377344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:2.0:1788575950.341046:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864377600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575950.341064:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.341067:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575950.341069:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.341075:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7700 00000100:00080000:2.0:1788575950.341079:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575950.341083:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575950.341088:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575950.341102:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864377728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575950.341111:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575950.341114:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575950.341117:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575950.341121:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7700 00000100:00080000:2.0:1788575950.341124:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575950.341130:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575950.341132:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575950.341140:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575950.341155:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864377856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575950.341195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff92fc4617c000 x1875449864377344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575950.341219:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864377344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575950.341231:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575950.341236:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575950.341240:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864377344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575950.341247:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575950.341283:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575950.341296:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff92fc4617fb80 x1875449864377472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575950.341306:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864377472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575950.341313:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575950.341316:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575950.341319:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575950.341356:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575950.341373:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575950.341381:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff92fc4617e680 x1875449864377600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575950.341395:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864377600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575950.341401:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575950.341403:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575950.341405:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575950.341409:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575950.341422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575950.341428:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff92fc4617c380 x1875449864377728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575950.341434:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864377728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575950.341439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575950.341441:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575950.341442:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575950.341446:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575950.341472:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575950.341479:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575950/real 1788575950] req@ffff92fc4617d180 x1875449864377856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575950.341487:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864377856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575950.341493:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575950.341495:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575950.341497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864377856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575950.788947:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575950.853110:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575951.812913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575951.878241:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575952.837022:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575952.900826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575953.864920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575953.924877:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575954.884933:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575954.948911:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575955.460827:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575955.460838:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575955.460842:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575955.460852:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7706 00000100:00080000:2.0:1788575955.460861:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575955.460869:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575955.460904:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864377984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575955.460935:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575955.460939:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575955.460941:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575955.460944:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7706 00000100:00080000:2.0:1788575955.460948:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575955.460952:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575955.460960:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575955.460972:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864378112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575955.460992:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575955.460995:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575955.460997:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575955.461001:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7706 00000100:00080000:2.0:1788575955.461004:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575955.461007:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575955.461012:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575955.461023:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864378240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575955.461032:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575955.461036:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575955.461037:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575955.461040:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7706 00000100:00080000:2.0:1788575955.461043:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575955.461046:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575955.461051:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575955.461059:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864378368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575955.461068:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575955.461071:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575955.461073:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575955.461076:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7706 00000100:00080000:2.0:1788575955.461079:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575955.461082:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575955.461086:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575955.461095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864378496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575955.461480:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864377984:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788575955.461604:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575955.461647:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff92fc4617f800 x1875449864377984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575955.461666:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864377984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575955.461676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575955.461680:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575955.461684:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864377984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575955.461691:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378112:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575955.461712:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575955.461750:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff92fc4617f100 x1875449864378112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575955.461763:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864378112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575955.461776:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575955.461780:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575955.461783:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575955.461789:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575955.461808:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575955.461820:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff92fc4617dc00 x1875449864378240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575955.461831:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864378240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575955.461840:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575955.461843:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575955.461846:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575955.461852:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575955.461872:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575955.461896:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff92fc4617ed80 x1875449864378368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575955.461908:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864378368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575955.461915:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575955.461918:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575955.461921:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575955.461927:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575955.461944:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575955.461953:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575955/real 1788575955] req@ffff92fc4617c700 x1875449864378496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575955.461962:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864378496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575955.461968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575955.461970:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575955.461973:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575955.908859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575955.972844:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575956.932935:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575956.996840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575957.956880:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575958.020835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575958.982985:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575959.045500:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575960.004896:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575960.068801:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575960.580872:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.580882:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575960.580886:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.580897:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7711 00000100:00080000:0.0:1788575960.580908:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.580914:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575960.580955:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864378624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.580981:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.580985:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.580987:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.580991:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7711 00000100:00080000:0.0:1788575960.580995:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.580999:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575960.581008:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.581020:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864378752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.581036:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.581053:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575960.581056:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575960.581060:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864378624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575960.581063:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7711 00000100:00080000:0.0:1788575960.581071:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.581075:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575960.581080:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.581093:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864378880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.581104:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.581108:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.581109:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.581113:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7711 00000100:00080000:0.0:1788575960.581117:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.581121:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575960.581125:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575960.581128:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788575960.581141:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864379008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.581170:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575960.581171:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff92fc6704a300 x1875449864378624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575960.581178:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575960.581182:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.581189:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7711 00000100:00080000:0.0:1788575960.581194:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575960.581195:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864378624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575960.581203:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:3.0:1788575960.581208:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788575960.581210:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575960.581213:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575960.581217:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864378624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575960.581223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788575960.581228:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864379136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575960.581248:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575960.581263:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff92fc6704aa00 x1875449864378752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575960.581276:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864378752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575960.581284:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575960.581287:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575960.581291:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575960.581313:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575960.581333:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575960.581345:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff92fc6704a680 x1875449864378880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575960.581352:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864378880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575960.581374:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575960.581376:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575960.581378:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864378880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575960.581382:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575960.581396:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575960.581403:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff92fc6704ad80 x1875449864379008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575960.581410:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864379008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575960.581414:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575960.581416:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575960.581418:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575960.581421:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575960.581430:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575960.581435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff92fc67049880 x1875449864379136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575960.581441:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864379136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575960.581446:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575960.581447:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575960.581449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575961.028963:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575961.092864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575962.052973:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575962.116821:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575963.078299:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575963.140837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575964.100873:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575964.164897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575965.124890:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575965.188832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575965.700842:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.700853:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575965.700857:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.700867:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7716 00000100:00080000:0.0:1788575965.700878:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.700885:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575965.700919:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864379264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575965.700953:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.700957:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575965.700959:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.700963:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7716 00000100:00080000:0.0:1788575965.700967:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.700971:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575965.700980:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575965.700993:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864379392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575965.701006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864379264:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575965.701012:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.701017:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575965.701019:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.701023:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7716 00000100:00080000:0.0:1788575965.701028:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.701032:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575965.701037:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575965.701049:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864379520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575965.701058:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.701061:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575965.701063:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.701067:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7716 00000100:00080000:0.0:1788575965.701072:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575965.701072:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575965.701079:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575965.701084:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575965.701096:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864379648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575965.701107:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575965.701110:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575965.701112:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575965.701119:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7716 00000100:00100000:1.0:1788575965.701121:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff92fc67049c00 x1875449864379264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575965.701124:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575965.701132:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575965.701140:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575965.701156:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864379776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575965.701158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864379264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.701180:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.701185:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575965.701190:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864379264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575965.701214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575965.701239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575965.701252:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff92fc67048e00 x1875449864379392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.701267:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864379392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.701276:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.701279:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575965.701282:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379392:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575965.701289:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575965.701322:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575965.701334:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff92fc67049180 x1875449864379520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.701346:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864379520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.701375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.701379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575965.701382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575965.701389:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379648:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575965.701410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575965.701421:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff92fc6704bb80 x1875449864379648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.701432:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864379648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.701441:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.701444:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575965.701448:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575965.701455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575965.701472:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575965.701483:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff92fc67049500 x1875449864379776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575965.701494:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864379776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575965.701502:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575965.701505:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575965.701508:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864379776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575966.148849:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575966.213292:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575967.173349:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575967.236850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575968.196887:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575968.260846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575969.220840:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575969.284832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575970.244866:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575970.308974:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575970.823465:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575970.823476:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575970.823480:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575970.823492:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7721 00000100:00080000:2.0:1788575970.823503:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575970.823510:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575970.823551:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864379904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575970.823579:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575970.823583:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575970.823586:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575970.823590:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7721 00000100:00080000:2.0:1788575970.823593:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575970.823597:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575970.823606:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575970.823617:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864380032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575970.823627:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575970.823630:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575970.823631:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575970.823635:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7721 00000100:00080000:2.0:1788575970.823638:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575970.823642:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575970.823647:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575970.823659:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864380160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575970.823676:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864379904:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575970.823678:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575970.823686:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575970.823688:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575970.823692:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7721 00000100:00080000:2.0:1788575970.823694:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575970.823698:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575970.823703:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575970.823715:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864380288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575970.823769:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575970.823814:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff92fc4617f480 x1875449864379904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.823833:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575970.823834:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864379904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.823841:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575970.823844:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575970.823852:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7721 00000100:00080000:1.0:1788575970.823854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575970.823857:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575970.823861:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788575970.823865:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575970.823870:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864379904:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788575970.823874:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575970.823881:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:2.0:1788575970.823890:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864380416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575970.823908:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575970.823922:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff92fc4617ca80 x1875449864380032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.823945:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864380032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.823953:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.823956:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575970.823959:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575970.823984:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575970.824005:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575970.824016:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff92fc4617c700 x1875449864380160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.824028:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864380160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.824035:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.824038:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575970.824041:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575970.824046:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575970.824064:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575970.824075:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff92fc4617ed80 x1875449864380288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.824085:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864380288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.824093:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.824096:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575970.824098:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575970.824102:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575970.824116:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575970.824122:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff92fc4617dc00 x1875449864380416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575970.824129:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864380416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575970.824133:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575970.824135:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575970.824137:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575971.268902:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575971.332897:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575972.293424:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575972.356842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575973.316877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575973.380874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575974.340905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575974.404839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575975.364926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575975.428819:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788575975.940840:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575975.940852:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575975.940856:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575975.940866:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7726 00000100:00080000:2.0:1788575975.940878:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575975.940885:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788575975.940931:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864380544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575975.940959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575975.940963:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575975.940965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575975.940975:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7726 00000100:00080000:2.0:1788575975.940979:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575975.940983:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575975.940993:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575975.941008:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864380672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575975.941016:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864380544:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788575975.941030:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575975.941035:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575975.941037:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575975.941041:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7726 00000100:00080000:2.0:1788575975.941045:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575975.941048:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575975.941053:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575975.941074:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864380800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575975.941084:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788575975.941087:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575975.941092:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575975.941094:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575975.941098:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7726 00000100:00080000:2.0:1788575975.941103:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575975.941107:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788575975.941112:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575975.941124:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864380928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575975.941132:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff92fc4617f100 x1875449864380544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.941137:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575975.941146:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575975.941157:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575975.941158:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864380544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.941170:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7726 00000100:00080000:1.0:1788575975.941173:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575975.941176:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575975.941180:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788575975.941185:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575975.941187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864380544:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788575975.941193:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575975.941208:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864381056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575975.941214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575975.941240:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575975.941253:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff92fc4617f800 x1875449864380672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.941263:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864380672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.941270:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.941273:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575975.941276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575975.941294:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575975.941312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575975.941322:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff92fc4617d180 x1875449864380800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.941334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864380800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.941341:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.941344:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575975.941356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575975.941362:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575975.941378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575975.941389:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff92fc4617c380 x1875449864380928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.941407:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864380928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.941418:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.941421:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575975.941424:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864380928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575975.941438:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575975.941456:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575975.941465:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff92fc4617e680 x1875449864381056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575975.941475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864381056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575975.941482:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575975.941485:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575975.941488:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575976.388891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575976.452849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575977.412923:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575977.476825:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575978.436886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575978.500848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575979.460899:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575979.524792:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575980.484874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575980.548884:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575981.060850:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575981.060860:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575981.060864:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575981.060873:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7731 00000100:00080000:0.0:1788575981.060883:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575981.060890:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575981.060923:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864381184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575981.060952:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575981.060955:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575981.060957:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575981.060960:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7731 00000100:00080000:0.0:1788575981.060962:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575981.060965:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575981.060973:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575981.060987:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864381312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575981.060997:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575981.061001:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575981.061003:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575981.061006:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864381184:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788575981.061008:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7731 00000100:00080000:0.0:1788575981.061014:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575981.061019:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575981.061023:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575981.061032:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864381440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575981.061038:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575981.061040:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575981.061042:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575981.061044:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7731 00000100:00080000:0.0:1788575981.061046:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575981.061048:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575981.061051:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575981.061060:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864381568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575981.061068:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575981.061071:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575981.061073:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575981.061078:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788575981.061079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7731 00000100:00080000:0.0:1788575981.061085:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575981.061089:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575981.061094:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575981.061105:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864381696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575981.061131:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575981/real 1788575981] req@ffff92fc6704b100 x1875449864381184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575981.061149:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864381184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575981.061160:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575981.061170:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575981.061174:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864381184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575981.061196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575981.061217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575981.061230:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575981/real 1788575981] req@ffff92fc67049500 x1875449864381312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575981.061242:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864381312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575981.061250:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575981.061253:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575981.061256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575981.061263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575981.061284:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575981.061291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575981/real 1788575981] req@ffff92fc6704bb80 x1875449864381440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575981.061302:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864381440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575981.061307:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575981.061309:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575981.061311:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575981.061314:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575981.061324:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575981.061329:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575981/real 1788575981] req@ffff92fc67049180 x1875449864381568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575981.061335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864381568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575981.061339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575981.061341:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575981.061343:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575981.061345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575981.061354:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575981.061359:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575981/real 1788575981] req@ffff92fc67048e00 x1875449864381696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575981.061365:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864381696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575981.061373:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575981.061374:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575981.061376:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575981.508915:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575981.572865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575982.532877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575982.596826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575983.556891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575983.620826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575984.580886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575984.644835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575985.604946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575985.668822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788575986.180855:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575986.180865:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575986.180869:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575986.180879:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7736 00000100:00080000:0.0:1788575986.180889:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575986.180897:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788575986.180961:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864381824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575986.181001:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575986.181006:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575986.181008:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575986.181012:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7736 00000100:00080000:0.0:1788575986.181016:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575986.181019:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575986.181030:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575986.181042:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864381952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575986.181051:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575986.181054:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575986.181056:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575986.181059:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7736 00000100:00080000:0.0:1788575986.181063:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575986.181066:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575986.181070:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575986.181081:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864382080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575986.181099:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575986.181102:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575986.181104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575986.181107:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7736 00000100:00080000:0.0:1788575986.181111:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575986.181114:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575986.181119:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575986.181129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864382208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575986.181137:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575986.181140:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575986.181142:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575986.181145:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7736 00000100:00080000:0.0:1788575986.181148:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575986.181151:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788575986.181156:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575986.181167:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864382336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575986.181304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864381824:192.168.202.132@tcp:250:kworker.0 00000800:00000100:1.0:1788575986.181376:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575986.181416:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff92fc67049c00 x1875449864381824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.181433:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864381824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.181444:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.181448:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575986.181451:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864381824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788575986.181458:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575986.181476:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575986.181487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff92fc67049880 x1875449864381952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.181499:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864381952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.181507:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.181510:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575986.181513:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864381952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575986.181518:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:1.0:1788575986.181536:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575986.181546:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff92fc6704ad80 x1875449864382080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.181556:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864382080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.181564:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.181566:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575986.181569:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575986.181575:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575986.181619:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575986.181630:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff92fc6704a680 x1875449864382208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.181641:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864382208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.181654:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.181667:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575986.181671:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575986.181677:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:1.0:1788575986.181695:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:1.0:1788575986.181704:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575986/real 1788575986] req@ffff92fc6704aa00 x1875449864382336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575986.181714:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864382336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575986.181744:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575986.181748:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575986.181751:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575986.628864:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575986.692829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575987.652972:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575987.716856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575988.676846:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575988.740842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575989.700909:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575989.764847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575990.724855:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575990.788865:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788575991.300873:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575991.300883:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575991.300887:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575991.300897:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7741 00000100:00080000:3.0:1788575991.300907:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575991.300914:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575991.300950:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864382464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575991.300975:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575991.300980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575991.300981:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575991.300985:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7741 00000100:00080000:3.0:1788575991.300988:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575991.300992:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575991.301002:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575991.301018:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864382464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575991.301028:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864382592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575991.301040:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575991.301043:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575991.301045:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575991.301048:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7741 00000100:00080000:3.0:1788575991.301052:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575991.301056:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575991.301060:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575991.301071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864382720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575991.301080:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575991.301084:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575991.301085:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575991.301089:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7741 00000100:00080000:3.0:1788575991.301092:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575991.301096:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575991.301102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575991.301115:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864382848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575991.301124:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575991.301127:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575991.301129:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575991.301133:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7741 00000100:00100000:1.0:1788575991.301136:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382592:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788575991.301138:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575991.301146:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788575991.301152:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575991.301165:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864382976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575991.301174:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575991.301201:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575991.301225:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788575991.748926:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575991.812868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575992.772909:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575992.836853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575993.796908:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575993.860849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:0.0:1788575994.373032:0:2380:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:0.0:1788575994.373071:0:2380:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575994.373087:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575994.373128:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575994.373139:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575994.373163:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:0.0:1788575994.373171:0:2380:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:1.0:1788575994.373234:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff92fc67338000 x1875449864382976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.373256:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864382976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.373267:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.373272:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575994.373276:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382976:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575994.373283:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff92fc6733a300 x1875449864382592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.373295:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864382592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.373302:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.373305:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575994.373309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575994.373314:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff92fc67339500 x1875449864382720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.373339:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864382720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.373346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.373350:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575994.373352:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:1.0:1788575994.373358:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff92fc6733aa00 x1875449864382848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.373369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864382848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.373376:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.373379:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575994.373382:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864382848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:1.0:1788575994.373388:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575991/real 1788575994] req@ffff92fc6733b480 x1875449864382464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575994.373398:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864382464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575994.373404:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575994.373407:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:1.0:1788575994.373411:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864382464:192.168.202.132@tcp:250:kworker.0 00010000:00010000:0.0:1788575994.820874:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575994.884820:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575995.844928:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575995.908836:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788575996.420839:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.420846:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575996.420849:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.420856:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7747 00000100:00080000:1.0:1788575996.420864:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.420869:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788575996.420896:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864383104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575996.420930:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.420933:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575996.420934:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.420937:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7747 00000100:00080000:1.0:1788575996.420940:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.420942:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575996.420951:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575996.420964:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864383232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575996.420969:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864383104:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788575996.420980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.420983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575996.420985:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.420992:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7747 00000100:00080000:1.0:1788575996.420996:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.420999:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788575996.421005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575996.421017:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864383360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575996.421027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.421030:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575996.421032:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.421036:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7747 00000100:00080000:1.0:1788575996.421039:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.421042:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:3.0:1788575996.421046:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788575996.421048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575996.421061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864383488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575996.421071:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575996.421074:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575996.421075:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575996.421079:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7747 00000100:00080000:1.0:1788575996.421083:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575996.421089:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788575996.421090:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff92fc67288380 x1875449864383104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575996.421095:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575996.421109:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864383616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575996.421116:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864383104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575996.421131:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575996.421135:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575996.421139:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864383104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788575996.421165:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383232:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575996.421188:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575996.421200:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff92fc67289c00 x1875449864383232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575996.421212:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864383232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575996.421220:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575996.421223:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575996.421226:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575996.421231:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383360:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788575996.421249:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575996.421261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff92fc6728a680 x1875449864383360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575996.421272:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864383360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575996.421279:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575996.421282:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575996.421285:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788575996.421290:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383488:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575996.421309:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575996.421319:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff92fc67289f80 x1875449864383488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575996.421335:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864383488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575996.421343:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575996.421346:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575996.421349:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788575996.421354:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383616:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788575996.421372:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788575996.421383:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575996/real 1788575996] req@ffff92fc6728b800 x1875449864383616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575996.421394:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864383616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575996.421402:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575996.421405:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788575996.421408:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788575996.868881:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575996.932849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575997.892885:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575997.956848:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788575998.916817:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575998.980845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788575999.940922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576000.004842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576000.964933:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576001.028805:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576001.540835:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576001.540845:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576001.540850:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576001.540860:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7752 00000100:00080000:1.0:1788576001.540870:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576001.540877:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576001.540910:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864383744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576001.540940:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576001.540943:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576001.540945:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576001.540948:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7752 00000100:00080000:1.0:1788576001.540952:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576001.540955:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576001.540964:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576001.540977:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864383872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576001.540987:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576001.540991:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576001.540994:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788576001.540996:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864383744:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788576001.541001:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7752 00000100:00080000:1.0:1788576001.541006:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576001.541010:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576001.541015:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576001.541025:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864384000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576001.541034:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576001.541037:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576001.541039:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576001.541043:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7752 00000100:00080000:1.0:1788576001.541053:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788576001.541071:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788576001.541073:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576001.541080:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576001.541091:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864384128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576001.541101:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576001.541109:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576001.541110:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576001.541114:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7752 00000100:00080000:1.0:1788576001.541117:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788576001.541121:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff92fc6728ad80 x1875449864383744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576001.541124:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576001.541132:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576001.541148:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864384256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576001.541156:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864383744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576001.541168:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576001.541171:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576001.541176:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864383744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576001.541196:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576001.541217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576001.541228:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff92fc67289880 x1875449864383872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576001.541240:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864383872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576001.541248:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576001.541251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576001.541254:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864383872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576001.541260:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576001.541274:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576001.541281:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff92fc6728b100 x1875449864384000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576001.541288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864384000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576001.541297:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576001.541299:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576001.541301:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576001.541304:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576001.541318:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576001.541327:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff92fc67288000 x1875449864384128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576001.541338:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864384128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576001.541346:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576001.541359:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576001.541363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576001.541369:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576001.541392:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576001.541406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576001/real 1788576001] req@ffff92fc67288e00 x1875449864384256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576001.541417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864384256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576001.541425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576001.541428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576001.541432:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576001.988958:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576002.052843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576003.012910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576003.076896:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576004.036907:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576004.100826:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576005.061416:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576005.124853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576006.085459:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576006.148812:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788576006.661031:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576006.661042:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576006.661046:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576006.661056:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7757 00000100:00080000:3.0:1788576006.661068:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576006.661075:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788576006.661110:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864384384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576006.661138:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576006.661142:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576006.661144:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576006.661148:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7757 00000100:00080000:3.0:1788576006.661151:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576006.661155:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576006.661164:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576006.661177:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864384512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576006.661188:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576006.661192:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788576006.661194:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864384384:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788576006.661195:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576006.661203:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7757 00000100:00080000:3.0:1788576006.661207:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576006.661222:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576006.661227:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576006.661239:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864384640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576006.661250:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576006.661253:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576006.661255:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576006.661258:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7757 00000100:00080000:3.0:1788576006.661263:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576006.661268:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576006.661273:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788576006.661275:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576006.661287:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864384768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576006.661297:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576006.661300:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576006.661302:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576006.661306:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7757 00000100:00080000:3.0:1788576006.661309:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576006.661313:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576006.661318:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff92fc67338380 x1875449864384384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788576006.661320:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576006.661336:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864384896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576006.661342:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864384384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576006.661357:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576006.661370:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576006.661375:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864384384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576006.661400:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576006.661422:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576006.661433:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff92fc67338a80 x1875449864384512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576006.661444:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864384512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576006.661452:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576006.661455:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576006.661458:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576006.661464:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576006.661484:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576006.661496:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff92fc6733b800 x1875449864384640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576006.661508:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864384640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576006.661516:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576006.661518:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576006.661522:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576006.661527:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576006.661623:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576006.661635:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff92fc67339880 x1875449864384768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576006.661647:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864384768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576006.661655:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576006.661658:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576006.661661:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576006.661667:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576006.661685:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576006.661705:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576006] req@ffff92fc67339c00 x1875449864384896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576006.661753:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864384896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576006.661762:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576006.661765:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576006.661768:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864384896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576007.108931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576007.172846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576008.132900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576008.197340:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576009.156884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576009.220986:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576010.180934:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576010.244841:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576011.204947:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576011.268898:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576011.782325:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576011.782335:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576011.782340:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576011.782350:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7762 00000100:00080000:1.0:1788576011.782361:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576011.782379:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576011.782419:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864385024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576011.782455:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576011.782459:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576011.782461:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576011.782470:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7762 00000100:00080000:1.0:1788576011.782475:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576011.782478:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576011.782490:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576011.782502:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864385152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576011.782524:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576011.782528:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576011.782530:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576011.782533:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7762 00000100:00080000:1.0:1788576011.782537:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576011.782540:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576011.782544:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576011.782554:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864385280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576011.782563:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576011.782566:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576011.782568:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576011.782571:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7762 00000100:00080000:1.0:1788576011.782574:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576011.782578:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576011.782582:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576011.782593:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864385408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576011.782709:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864385024:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576011.782823:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576011.782866:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff92fc6728a300 x1875449864385024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576011.782897:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864385024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576011.782907:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576011.782911:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576011.782914:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864385024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576011.782922:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576011.782954:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576011.782966:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff92fc6728b480 x1875449864385152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576011.782983:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864385152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576011.782991:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576011.782993:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576011.782996:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576011.783002:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576011.783020:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576011.783029:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff92fc6728aa00 x1875449864385280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576011.783039:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864385280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576011.783046:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576011.783048:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576011.783051:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576011.783679:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576011.783707:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788576011.783751:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576011.783756:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576011.783758:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576011.783763:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7762 00000100:00080000:1.0:1788576011.783768:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576011.783773:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576011.783780:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576011.783795:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864385536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576011.783849:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff92fc67288e00 x1875449864385408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576011.783862:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864385408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576011.783876:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576011.783890:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576011.783894:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576011.783907:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576011.783937:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576011.783950:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff92fc67288000 x1875449864385536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576011.783961:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864385536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576011.783968:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576011.783971:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576011.783974:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576012.229435:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576012.292842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576013.252939:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576013.316816:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576014.276927:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576014.340840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576015.300920:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576015.364854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576016.324883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576016.388839:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576016.900861:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576016.900868:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576016.900872:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576016.900879:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7767 00000100:00080000:2.0:1788576016.900887:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576016.900892:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576016.900923:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864385664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576016.900941:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576016.900943:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576016.900944:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576016.900947:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7767 00000100:00080000:2.0:1788576016.900949:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576016.900951:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576016.900961:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576016.900969:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864385792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576016.900976:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576016.900980:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788576016.900981:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864385664:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788576016.900982:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576016.900989:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7767 00000100:00080000:2.0:1788576016.900991:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576016.900993:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576016.900996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576016.901003:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864385920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576016.901009:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576016.901011:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576016.901013:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576016.901016:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7767 00000100:00080000:2.0:1788576016.901020:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576016.901024:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576016.901028:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788576016.901034:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788576016.901041:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864386048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576016.901051:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576016.901055:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576016.901057:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576016.901061:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7767 00000100:00080000:2.0:1788576016.901065:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788576016.901066:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff92fc4617fb80 x1875449864385664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576016.901069:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576016.901076:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576016.901087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864386176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576016.901094:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864385664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.901106:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.901111:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576016.901116:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864385664:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576016.901139:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576016.901160:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576016.901171:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff92fc4617c000 x1875449864385792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.901183:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864385792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.901191:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.901194:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576016.901197:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576016.901203:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576016.901222:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576016.901232:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff92fc4617d880 x1875449864385920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.901244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864385920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.901251:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.901254:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576016.901257:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864385920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576016.901263:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576016.901281:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576016.901291:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff92fc4617ea00 x1875449864386048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.901301:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864386048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.901308:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.901311:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576016.901314:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576016.901320:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576016.901338:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576016.901348:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff92fc4617e300 x1875449864386176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576016.901367:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864386176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576016.901378:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576016.901382:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576016.901384:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576017.348859:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576017.412854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576018.372912:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576018.437458:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576019.396914:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576019.461052:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576020.420968:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576020.485763:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576021.444888:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576021.508859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576022.021187:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576022.021198:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576022.021203:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576022.021213:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7772 00000100:00080000:2.0:1788576022.021224:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576022.021231:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576022.021268:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864386304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576022.021299:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576022.021305:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576022.021307:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576022.021311:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7772 00000100:00080000:2.0:1788576022.021314:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576022.021318:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576022.021327:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576022.021340:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864386432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576022.021351:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576022.021355:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576022.021356:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576022.021361:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7772 00000100:00100000:0.0:1788576022.021362:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864386304:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788576022.021365:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576022.021370:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576022.021377:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576022.021397:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864386560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576022.021405:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576022.021408:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576022.021410:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576022.021414:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7772 00000100:00080000:2.0:1788576022.021417:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576022.021421:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576022.021426:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576022.021436:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864386688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788576022.021446:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788576022.021448:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576022.021453:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576022.021455:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576022.021460:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7772 00000100:00080000:2.0:1788576022.021463:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576022.021468:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576022.021473:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576022.021484:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864386816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576022.021502:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576022/real 1788576022] req@ffff92fc4617ce00 x1875449864386304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576022.021521:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864386304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576022.021532:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576022.021536:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576022.021540:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864386304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576022.021564:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576022.021586:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576022.021598:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576022/real 1788576022] req@ffff92fc4617d500 x1875449864386432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576022.021609:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864386432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576022.021623:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576022.021626:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576022.021629:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576022.021635:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576022.021652:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576022.021662:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576022/real 1788576022] req@ffff92fc4617e300 x1875449864386560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576022.021695:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864386560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576022.021709:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576022.021713:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576022.021716:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576022.021753:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576022.021773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576022.021783:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576022/real 1788576022] req@ffff92fc4617ea00 x1875449864386688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576022.021794:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864386688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576022.021802:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576022.021805:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576022.021808:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576022.021814:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576022.021839:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576022.021850:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576022/real 1788576022] req@ffff92fc4617d880 x1875449864386816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576022.021861:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864386816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576022.021868:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576022.021870:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576022.021874:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864386816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576022.469003:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576022.532838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576023.492961:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576023.557008:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576024.516947:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576024.580842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576025.540842:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576025.604872:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576026.564933:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576026.628979:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576027.140783:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576027.140793:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576027.140798:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576027.140808:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7777 00000100:00080000:2.0:1788576027.140821:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576027.140847:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576027.140891:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864386944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576027.140921:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576027.140925:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576027.140927:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576027.140931:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7777 00000100:00080000:2.0:1788576027.140936:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576027.140940:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576027.140950:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576027.140964:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864387072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576027.140973:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576027.140977:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576027.140979:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788576027.140980:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864386944:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788576027.140983:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7777 00000100:00080000:2.0:1788576027.140989:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576027.140992:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576027.140995:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576027.141011:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864387200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576027.141021:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576027.141025:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576027.141027:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576027.141031:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7777 00000100:00080000:2.0:1788576027.141035:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576027.141039:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576027.141044:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788576027.141050:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788576027.141057:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864387328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576027.141071:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576027.141074:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576027.141076:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576027.141079:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7777 00000100:00080000:2.0:1788576027.141083:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576027.141087:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576027.141091:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576027.141095:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576027/real 1788576027] req@ffff92fc4617c000 x1875449864386944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576027.141104:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864387456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576027.141119:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864386944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576027.141131:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576027.141135:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576027.141139:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864386944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576027.141164:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576027.141184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576027.141195:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576027/real 1788576027] req@ffff92fc4617fb80 x1875449864387072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576027.141205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864387072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576027.141212:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576027.141215:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576027.141218:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576027.141223:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576027.141239:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576027.141247:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576027/real 1788576027] req@ffff92fc4617e680 x1875449864387200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576027.141257:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864387200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576027.141263:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576027.141266:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576027.141269:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576027.141277:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576027.141291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576027.141298:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576027/real 1788576027] req@ffff92fc4617c380 x1875449864387328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576027.141308:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864387328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576027.141315:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576027.141317:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576027.141320:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576027.141324:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576027.141336:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576027.141343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576027/real 1788576027] req@ffff92fc4617d180 x1875449864387456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576027.141358:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864387456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576027.141372:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576027.141374:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576027.141377:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576027.589325:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576027.652832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576028.613049:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576028.676842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576029.636931:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576029.700892:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576030.660916:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576030.724843:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576031.684923:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576031.748863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576032.260864:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.260875:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576032.260878:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.260888:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7782 00000100:00080000:1.0:1788576032.260898:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.260905:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576032.260941:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864387584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.260961:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.260963:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576032.260964:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.260967:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7782 00000100:00080000:1.0:1788576032.260969:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.260972:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576032.260978:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576032.261003:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864387712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.261013:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.261016:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576032.261018:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.261032:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7782 00000100:00100000:0.0:1788576032.261033:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864387584:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788576032.261037:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.261043:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576032.261048:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576032.261062:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864387840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.261072:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.261075:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576032.261077:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.261081:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7782 00000100:00080000:1.0:1788576032.261084:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.261088:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788576032.261093:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788576032.261094:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576032.261107:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864387968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576032.261117:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576032.261120:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576032.261122:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576032.261126:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7782 00000100:00080000:1.0:1788576032.261129:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576032.261132:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576032.261136:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576032.261144:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864388096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576032.261148:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff92fc6728b100 x1875449864387584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.261169:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864387584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.261179:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.261183:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576032.261187:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864387584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576032.261291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576032.261312:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576032.261323:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff92fc67289880 x1875449864387712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.261341:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864387712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.261349:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.261352:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576032.261356:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576032.261385:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576032.261401:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576032.261416:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff92fc6728ad80 x1875449864387840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.261427:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864387840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.261435:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.261437:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576032.261440:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576032.261445:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576032.261459:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576032.261466:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff92fc6728b800 x1875449864387968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.261476:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864387968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.261489:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.261492:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576032.261495:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864387968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576032.261500:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576032.261512:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576032.261519:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576032/real 1788576032] req@ffff92fc67289f80 x1875449864388096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576032.261529:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864388096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576032.261536:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576032.261538:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576032.261541:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576032.711265:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576032.773359:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576033.732812:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576033.796867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576034.756893:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576034.820847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576035.781122:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576035.846387:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576036.804903:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576036.868863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576037.380830:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576037.380841:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576037.380845:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576037.380854:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7787 00000100:00080000:1.0:1788576037.380865:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576037.380872:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576037.380906:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864388224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576037.380934:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576037.380937:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576037.380939:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576037.380942:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7787 00000100:00080000:1.0:1788576037.380945:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576037.380949:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576037.380958:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576037.380970:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864388352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576037.380980:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788576037.380983:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864388224:192.168.202.132@tcp:250:kworker.0 00000100:00080000:1.0:1788576037.380984:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576037.380988:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576037.380994:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7787 00000100:00080000:1.0:1788576037.380998:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576037.381003:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576037.381008:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576037.381021:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864388480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576037.381031:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576037.381034:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576037.381037:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576037.381041:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7787 00000100:00080000:1.0:1788576037.381044:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576037.381050:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000800:00000100:0.0:1788576037.381055:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000008:00100000:1.0:1788576037.381055:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576037.381069:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864388608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576037.381078:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576037.381082:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576037.381084:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576037.381087:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7787 00000100:00080000:1.0:1788576037.381091:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576037.381097:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576037.381098:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff92fc6728a680 x1875449864388224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788576037.381102:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576037.381118:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864388736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576037.381126:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864388224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.381162:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576037.381167:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576037.381171:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864388224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576037.381198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576037.381217:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576037.381227:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff92fc67289c00 x1875449864388352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576037.381238:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864388352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.381246:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576037.381249:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576037.381252:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576037.381257:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576037.381270:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576037.381277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff92fc67288380 x1875449864388480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576037.381288:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864388480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.381295:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576037.381298:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576037.381300:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576037.381306:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576037.381319:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576037.381326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff92fc6728bb80 x1875449864388608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576037.381337:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864388608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.381344:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576037.381347:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576037.381350:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576037.381355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576037.381368:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576037.381377:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576037/real 1788576037] req@ffff92fc67288700 x1875449864388736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576037.381387:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864388736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576037.381394:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576037.381398:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576037.381401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576037.828897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576037.892861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576038.852883:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576038.916859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576039.876911:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576039.940973:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576040.900999:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576040.964847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576041.924876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576041.988855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576042.501061:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576042.501069:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576042.501073:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576042.501080:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7793 00000100:00080000:1.0:1788576042.501089:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576042.501095:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576042.501129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288a80 x1875449864388864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576042.501160:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576042.501164:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576042.501166:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576042.501170:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7793 00000100:00080000:1.0:1788576042.501174:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576042.501177:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576042.501186:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576042.501211:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289500 x1875449864388992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576042.501221:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576042.501225:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576042.501226:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576042.501230:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7793 00000100:00080000:1.0:1788576042.501233:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576042.501237:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576042.501241:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576042.501252:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289180 x1875449864389120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576042.501260:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576042.501263:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576042.501265:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576042.501268:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7793 00000100:00080000:1.0:1788576042.501271:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576042.501275:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576042.501279:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576042.501291:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864389248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576042.501299:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576042.501301:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576042.501303:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576042.501307:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7793 00000100:00080000:1.0:1788576042.501310:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576042.501313:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576042.501317:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576042.501327:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864389376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576042.502258:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864388864:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576042.502315:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576042.502352:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff92fc67288a80 x1875449864388864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576042.502369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288a80 x1875449864388864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576042.502379:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576042.502383:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576042.502386:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864388864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576042.502393:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576042.502410:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576042.502419:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff92fc67289500 x1875449864388992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576042.502430:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289500 x1875449864388992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576042.502439:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576042.502442:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576042.502445:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864388992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576042.502450:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576042.502465:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576042.502473:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff92fc67289180 x1875449864389120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576042.502484:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289180 x1875449864389120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576042.502491:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576042.502494:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576042.502497:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576042.502503:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576042.502517:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576042.502525:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff92fc67288700 x1875449864389248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576042.502535:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864389248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576042.502542:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576042.502544:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576042.502548:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576042.502553:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576042.502566:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576042.502574:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576042/real 1788576042] req@ffff92fc6728bb80 x1875449864389376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576042.502591:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864389376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576042.502598:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576042.502601:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576042.502604:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576042.948911:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576043.012863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576043.973392:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576044.036852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576044.997186:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576045.061077:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576046.020930:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576046.085460:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576047.045274:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576047.108899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576047.620907:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.620919:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576047.620925:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.620937:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7798 00000100:00080000:0.0:1788576047.620950:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.620964:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576047.621014:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864389504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.621046:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.621050:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576047.621052:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.621056:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7798 00000100:00080000:0.0:1788576047.621060:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.621064:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576047.621076:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576047.621087:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864389632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.621097:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.621100:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576047.621101:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.621104:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7798 00000100:00080000:0.0:1788576047.621108:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788576047.621110:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864389504:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788576047.621113:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576047.621120:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576047.621131:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864389760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.621139:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.621142:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576047.621144:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.621147:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7798 00000100:00080000:0.0:1788576047.621150:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576047.621153:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576047.621157:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576047.621166:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864389888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576047.621178:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576047.621181:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576047.621183:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576047.621185:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7798 00000100:00080000:0.0:1788576047.621189:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788576047.621194:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788576047.621195:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576047.621202:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576047.621241:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864390016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576047.621275:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff92fc6704a300 x1875449864389504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576047.621294:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864389504/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576047.621303:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576047.621306:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576047.621309:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864389504:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576047.621334:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389632:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576047.621353:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576047.621362:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff92fc67049f80 x1875449864389632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576047.621369:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864389632/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576047.621375:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576047.621377:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576047.621380:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389632:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576047.621383:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389760:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576047.621394:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576047.621401:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff92fc67048700 x1875449864389760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576047.621413:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864389760/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576047.621420:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576047.621423:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576047.621426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389760:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576047.621432:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389888:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576047.621453:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576047.621465:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff92fc67048380 x1875449864389888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576047.621477:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864389888/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576047.621482:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576047.621484:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576047.621487:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864389888:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576047.621494:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390016:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576047.621513:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576047.621524:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff92fc67048000 x1875449864390016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576047.621536:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864390016/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576047.621543:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576047.621545:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576047.621547:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390016:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576048.068919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576048.133318:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576049.092922:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576049.156916:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576050.118779:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576050.180851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576051.140918:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576051.204867:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576052.164910:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576052.228818:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576052.740843:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576052.740853:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576052.740856:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576052.740866:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7803 00000100:00080000:2.0:1788576052.740876:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576052.740883:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576052.740917:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864390144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576052.740942:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576052.740946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576052.740947:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576052.740951:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7803 00000100:00080000:2.0:1788576052.740954:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576052.740958:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576052.740967:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576052.740982:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864390272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576052.740982:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864390144:192.168.202.132@tcp:250:kworker.0 00000100:00080000:2.0:1788576052.740997:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576052.741000:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576052.741003:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576052.741015:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7803 00000100:00080000:2.0:1788576052.741025:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576052.741029:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576052.741034:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788576052.741037:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:2.0:1788576052.741050:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864390400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576052.741060:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576052.741063:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576052.741065:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576052.741069:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7803 00000100:00080000:2.0:1788576052.741073:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788576052.741076:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff92fc4617f800 x1875449864390144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576052.741080:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576052.741089:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576052.741098:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864390144/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576052.741103:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864390528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576052.741125:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576052.741129:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576052.741132:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576052.741132:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576052.741137:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7803 00000100:00080000:2.0:1788576052.741142:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576052.741142:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00080000:2.0:1788576052.741150:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788576052.741152:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864390144:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788576052.741157:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576052.741169:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864390656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576052.741180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390272:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576052.741207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576052.741235:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff92fc4617f100 x1875449864390272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576052.741244:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864390272/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576052.741249:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576052.741251:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576052.741253:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390272:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576052.741256:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390400:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576052.741270:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576052.741277:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff92fc4617dc00 x1875449864390400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576052.741284:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864390400/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576052.741289:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576052.741291:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576052.741292:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390400:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576052.741296:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390528:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576052.741306:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576052.741311:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff92fc4617ed80 x1875449864390528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576052.741318:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864390528/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576052.741322:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576052.741324:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576052.741326:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390528:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576052.741335:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390656:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576052.741345:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576052.741350:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff92fc4617c700 x1875449864390656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576052.741356:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864390656/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576052.741360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576052.741362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576052.741363:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390656:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576053.188901:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576053.252781:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576054.212927:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576054.276852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576055.236946:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576055.300804:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576056.260848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576056.324840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576057.284889:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576057.348854:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576057.860851:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576057.860861:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576057.860867:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576057.860887:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7808 00000100:00080000:0.0:1788576057.860901:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576057.860908:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576057.860944:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864390784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576057.860977:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576057.860981:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576057.860983:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576057.860986:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7808 00000100:00080000:0.0:1788576057.860989:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576057.860993:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576057.861003:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576057.861017:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864390912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576057.861027:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864390784:192.168.202.132@tcp:250:kworker.0 00000100:00080000:0.0:1788576057.861029:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576057.861036:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576057.861038:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576057.861042:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7808 00000100:00080000:0.0:1788576057.861045:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576057.861049:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576057.861054:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576057.861071:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864391040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576057.861085:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576057.861089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576057.861091:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576057.861094:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7808 00000100:00080000:0.0:1788576057.861098:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576057.861102:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576057.861107:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576057.861119:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864391168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576057.861128:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576057.861131:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576057.861133:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576057.861137:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7808 00000100:00080000:0.0:1788576057.861141:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788576057.861142:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390912:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788576057.861148:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576057.861156:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576057.861169:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864391296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576057.861180:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576057.861205:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576057.861240:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391296:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576058.309156:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576058.372856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576059.333142:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576059.398786:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576060.356875:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576060.420840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:2.0:1788576060.933032:0:2381:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:2.0:1788576060.933075:0:2381:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788576060.933090:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788576060.933128:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788576060.933136:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788576060.933143:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:2.0:1788576060.933149:0:2381:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788576060.933347:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff92fc67049f80 x1875449864391296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576060.933384:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864391296/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576060.933396:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576060.933401:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576060.933406:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391296:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576060.933414:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff92fc67048000 x1875449864390912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576060.933426:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864390912/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576060.933434:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576060.933438:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576060.933441:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864390912:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576060.933448:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff92fc67048380 x1875449864391040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576060.933456:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864391040/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576060.933464:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576060.933468:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576060.933471:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391040:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576060.933477:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff92fc67048700 x1875449864391168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576060.933488:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864391168/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576060.933494:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576060.933497:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576060.933500:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391168:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576060.933506:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576060] req@ffff92fc67048a80 x1875449864390784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576060.933516:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864390784/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576060.933523:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576060.933526:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576060.933529:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864390784:192.168.202.132@tcp:250:kworker.0 00010000:00010000:2.0:1788576061.382219:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576061.444852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576062.405213:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576062.468840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576062.980973:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576062.980987:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576062.980992:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576062.981002:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7813 00000100:00080000:1.0:1788576062.981013:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576062.981019:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576062.981112:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864391424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576062.981198:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864391424:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576062.981271:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:1.0:1788576062.981301:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576062.981304:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576062.981306:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576062.981310:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7813 00000100:00080000:1.0:1788576062.981314:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576062.981318:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576062.981327:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576062.981340:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864391552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576062.984548:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff92fc67288380 x1875449864391424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.984577:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864391424/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.984587:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.984592:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576062.984596:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864391424:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576062.984628:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391552:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576062.984685:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576062.984703:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff92fc67289c00 x1875449864391552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.984714:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864391552/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.984739:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.984743:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576062.984746:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391552:192.168.202.132@tcp:38:kworker.0 00000100:00080000:1.0:1788576062.985700:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576062.985707:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576062.985710:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576062.985717:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7813 00000100:00080000:1.0:1788576062.985740:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576062.985748:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576062.985762:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576062.985790:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864391680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576062.985813:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576062.985817:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576062.985818:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576062.985822:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7813 00000100:00080000:1.0:1788576062.985825:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576062.985828:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576062.985834:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576062.985854:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864391808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576062.985863:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576062.985866:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576062.985867:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576062.985870:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7813 00000100:00080000:1.0:1788576062.985874:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576062.985877:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576062.985882:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576062.985891:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864391936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576062.986110:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391680:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576062.986154:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576062.986174:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff92fc6728a680 x1875449864391680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.986188:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864391680/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.986202:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.986205:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576062.986208:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391680:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576062.986214:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391808:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576062.986234:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576062.986256:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff92fc67289f80 x1875449864391808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.986266:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864391808/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.986273:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.986276:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576062.986279:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391808:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576062.986284:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391936:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576062.986302:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576062.986313:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff92fc6728b800 x1875449864391936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.986323:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864391936/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.986331:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.986333:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576062.986337:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864391936:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576063.428948:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576063.492969:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576064.452892:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576064.517188:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576065.476949:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576065.543483:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576066.502213:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576066.564840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576067.525386:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576067.589514:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576068.100876:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576068.100889:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576068.100895:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576068.100907:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7818 00000100:00080000:1.0:1788576068.100930:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576068.100942:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576068.100986:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864392064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576068.101029:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576068.101033:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576068.101035:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576068.101040:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7818 00000100:00080000:1.0:1788576068.101043:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576068.101047:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576068.101059:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576068.101073:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864392192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576068.101084:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576068.101087:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576068.101088:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576068.101092:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7818 00000100:00080000:1.0:1788576068.101108:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576068.101113:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576068.101118:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576068.101129:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864392320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576068.101139:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576068.101143:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576068.101145:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576068.101148:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7818 00000100:00080000:1.0:1788576068.101152:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576068.101156:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576068.101161:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576068.101172:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864392448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576068.101180:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576068.101183:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576068.101184:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576068.101187:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7818 00000100:00080000:1.0:1788576068.101190:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576068.101205:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576068.101210:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576068.101220:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864392576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576068.101364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864392064:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576068.101442:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576068.101504:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576068/real 1788576068] req@ffff92fc6728ad80 x1875449864392064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576068.101519:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864392064/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576068.101528:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576068.101531:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576068.101534:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864392064:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576068.101539:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392192:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576068.101555:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576068.101564:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576068/real 1788576068] req@ffff92fc67289880 x1875449864392192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576068.101571:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864392192/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576068.101576:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576068.101577:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576068.101579:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392192:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576068.101582:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392320:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576068.101593:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576068.101599:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576068/real 1788576068] req@ffff92fc6728b100 x1875449864392320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576068.101605:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864392320/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576068.101610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576068.101611:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576068.101613:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392320:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576068.101616:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392448:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576068.101628:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576068.101633:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576068/real 1788576068] req@ffff92fc67288000 x1875449864392448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576068.101640:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864392448/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576068.101644:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576068.101646:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576068.101647:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392448:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576068.101650:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392576:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576068.101660:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576068.101665:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576068/real 1788576068] req@ffff92fc67288e00 x1875449864392576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576068.101671:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864392576/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576068.101676:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576068.101678:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576068.101680:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392576:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576068.549124:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576068.612989:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576069.573081:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576069.636832:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576070.596879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576070.662383:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576071.623421:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576071.684829:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576072.644913:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576072.708870:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576073.221748:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576073.221763:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576073.221769:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576073.221782:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7823 00000100:00080000:0.0:1788576073.221795:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576073.221805:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576073.221853:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864392704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576073.221880:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576073.221884:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576073.221885:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576073.221890:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7823 00000100:00080000:0.0:1788576073.221893:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576073.221896:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576073.221906:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576073.221914:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864392832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576073.221920:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576073.221922:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576073.221923:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576073.221926:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7823 00000100:00080000:0.0:1788576073.221927:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576073.221929:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576073.221931:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576073.221940:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864392960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576073.221945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576073.221948:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576073.221949:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576073.221951:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7823 00000100:00080000:0.0:1788576073.221953:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576073.221955:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576073.221957:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576073.221962:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864393088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576073.221969:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576073.221971:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576073.221972:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576073.221974:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7823 00000100:00080000:0.0:1788576073.221975:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576073.221977:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576073.221979:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576073.221984:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864393216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576073.222889:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864392704:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576073.222968:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576073.223015:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff92fc6704a300 x1875449864392704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576073.223036:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864392704/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576073.223049:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576073.223054:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576073.223057:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864392704:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576073.223065:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392832:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576073.223087:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576073.223098:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff92fc6704aa00 x1875449864392832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576073.223109:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864392832/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576073.223117:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576073.223121:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576073.223124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392832:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576073.223130:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392960:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576073.223149:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576073.223159:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff92fc6704a680 x1875449864392960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576073.223170:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864392960/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576073.223178:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576073.223181:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576073.223185:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864392960:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576073.223191:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393088:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576073.223211:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576073.223222:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff92fc6704ad80 x1875449864393088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576073.223233:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864393088/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576073.223241:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576073.223244:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576073.223247:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393088:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576073.223254:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393216:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576073.223287:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576073.223299:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576073/real 1788576073] req@ffff92fc67049880 x1875449864393216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576073.223312:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864393216/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576073.223393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576073.223397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576073.223400:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393216:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576073.669181:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576073.732917:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576074.692878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576074.756838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576075.716868:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576075.780850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576076.740860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576076.804850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576077.766155:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576077.829487:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576078.340864:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576078.340874:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576078.340878:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576078.340888:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7828 00000100:00080000:0.0:1788576078.340899:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576078.340906:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576078.340944:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864393344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576078.340978:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576078.340983:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576078.340985:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576078.340989:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7828 00000100:00080000:0.0:1788576078.340992:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576078.340995:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576078.341005:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576078.341017:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864393472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576078.341027:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576078.341030:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576078.341032:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576078.341035:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7828 00000100:00080000:0.0:1788576078.341039:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576078.341042:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576078.341046:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576078.341057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864393600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576078.341066:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576078.341069:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576078.341071:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576078.341074:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7828 00000100:00080000:0.0:1788576078.341078:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576078.341081:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576078.341087:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576078.341099:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864393728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576078.341107:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576078.341110:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576078.341112:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576078.341115:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7828 00000100:00080000:0.0:1788576078.341118:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576078.341121:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576078.341125:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576078.341134:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864393856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576078.341373:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864393344:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576078.341432:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576078.341481:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576078/real 1788576078] req@ffff92fc67049c00 x1875449864393344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576078.341498:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864393344/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576078.341509:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576078.341513:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576078.341517:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864393344:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576078.341524:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393472:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576078.341543:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576078.341554:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576078/real 1788576078] req@ffff92fc67048e00 x1875449864393472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576078.341565:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864393472/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576078.341579:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576078.341582:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576078.341585:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393472:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576078.341590:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393600:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576078.341606:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576078.341615:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576078/real 1788576078] req@ffff92fc67049180 x1875449864393600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576078.341626:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864393600/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576078.341633:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576078.341636:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576078.341639:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393600:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576078.341644:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393728:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576078.341661:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576078.341670:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576078/real 1788576078] req@ffff92fc6704bb80 x1875449864393728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576078.341680:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864393728/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576078.341686:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576078.341689:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576078.341692:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393728:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576078.341698:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393856:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576078.341714:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576078.341745:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576078/real 1788576078] req@ffff92fc67049500 x1875449864393856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576078.341756:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864393856/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576078.341763:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576078.341766:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576078.341768:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864393856:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576078.789023:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576078.853689:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576079.814348:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576079.878447:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576080.837006:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576080.900853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576081.861551:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576081.924845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576082.885514:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576082.948977:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788576083.460846:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576083.460856:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576083.460860:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576083.460870:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7834 00000100:00080000:3.0:1788576083.460880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576083.460887:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788576083.460922:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339180 x1875449864393984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576083.460945:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576083.460949:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576083.460951:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576083.460955:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7834 00000100:00080000:3.0:1788576083.460960:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576083.460975:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576083.460985:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576083.460988:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864393984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576083.461000:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733ad80 x1875449864394112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576083.461011:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576083.461015:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576083.461017:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576083.461021:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7834 00000100:00080000:3.0:1788576083.461025:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576083.461029:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576083.461034:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788576083.461047:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576083.461061:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339f80 x1875449864394240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576083.461072:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576083.461075:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576083.461077:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576083.461081:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7834 00000100:00100000:0.0:1788576083.461084:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576083/real 1788576083] req@ffff92fc67339180 x1875449864393984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576083.461086:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576083.461095:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576083.461103:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339180 x1875449864393984/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788576083.461113:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788576083.461116:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576083.461120:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576083.461124:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864393984:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576083.461132:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339c00 x1875449864394368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576083.461143:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788576083.461146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394112:192.168.202.132@tcp:38:kworker.0 00000100:00080000:3.0:1788576083.461156:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576083.461158:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788576083.461167:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:3.0:1788576083.461167:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7834 00000100:00080000:3.0:1788576083.461175:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576083.461180:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576083.461185:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576083/real 1788576083] req@ffff92fc6733ad80 x1875449864394112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788576083.461186:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576083.461202:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339880 x1875449864394496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576083.461205:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733ad80 x1875449864394112/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576083.461225:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576083.461228:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576083.461232:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394112:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576083.461236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394240:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576083.461252:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576083.461261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576083/real 1788576083] req@ffff92fc67339f80 x1875449864394240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576083.461268:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339f80 x1875449864394240/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576083.461274:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576083.461275:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576083.461277:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394240:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576083.461281:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394368:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576083.461291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576083.461326:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576083/real 1788576083] req@ffff92fc67339c00 x1875449864394368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576083.461334:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339c00 x1875449864394368/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576083.461338:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576083.461340:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576083.461342:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394368:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576083.461353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394496:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576083.461366:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576083.461376:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576083/real 1788576083] req@ffff92fc67339880 x1875449864394496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576083.461386:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339880 x1875449864394496/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576083.461393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576083.461396:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576083.461399:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394496:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576083.909234:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576083.972859:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576084.932878:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576084.997433:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576085.957441:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576086.020856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576086.980967:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576087.045514:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576088.005491:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576088.068887:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788576088.581345:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.581355:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576088.581359:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.581370:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7839 00000100:00080000:3.0:1788576088.581381:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.581388:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788576088.581424:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b800 x1875449864394624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576088.581461:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.581465:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576088.581467:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.581471:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7839 00000100:00080000:3.0:1788576088.581474:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.581478:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576088.581488:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576088.581504:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338a80 x1875449864394752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576088.581514:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.581517:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576088.581519:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.581523:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7839 00000100:00080000:3.0:1788576088.581526:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.581530:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576088.581535:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576088.581545:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338380 x1875449864394880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576088.581553:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.581556:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576088.581558:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.581562:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7839 00000100:00080000:3.0:1788576088.581565:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.581569:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576088.581573:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576088.581584:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b480 x1875449864395008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576088.581607:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864394624:192.168.202.132@tcp:250:kworker.0 00000100:00080000:3.0:1788576088.581611:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.581617:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576088.581618:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.581622:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7839 00000100:00080000:3.0:1788576088.581627:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.581631:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576088.581637:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576088.581650:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733aa00 x1875449864395136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788576088.581664:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576088.581697:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff92fc6733b800 x1875449864394624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.581710:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b800 x1875449864394624/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.581753:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.581756:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576088.581759:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864394624:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576088.581764:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394752:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576088.581780:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576088.581788:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff92fc67338a80 x1875449864394752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.581795:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338a80 x1875449864394752/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.581801:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.581803:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576088.581804:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394752:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576088.581807:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394880:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576088.581817:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576088.581823:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff92fc67338380 x1875449864394880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.581858:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338380 x1875449864394880/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.581863:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.581865:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576088.581867:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864394880:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576088.581885:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395008:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576088.581897:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576088.581903:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff92fc6733b480 x1875449864395008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.581910:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b480 x1875449864395008/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.581923:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.581925:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576088.581927:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395008:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576088.581930:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395136:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576088.581945:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576088.581951:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff92fc6733aa00 x1875449864395136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.581957:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733aa00 x1875449864395136/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.581965:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.581967:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576088.581969:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395136:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576089.029012:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576089.092860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576090.052992:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576090.116987:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576091.076985:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576091.141157:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576092.101232:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576092.165475:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576093.124962:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576093.188894:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576093.701035:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576093.701044:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576093.701047:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576093.701057:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7844 00000100:00080000:0.0:1788576093.701067:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576093.701074:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576093.701111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704b100 x1875449864395264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576093.701169:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864395264:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576093.701219:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788576093.701245:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576093.701248:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576093.701250:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576093.701252:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7844 00000100:00080000:0.0:1788576093.701255:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576093.701258:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576093.701265:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576093.701276:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049500 x1875449864395392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576093.703565:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff92fc6704b100 x1875449864395264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576093.703589:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704b100 x1875449864395264/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576093.703600:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576093.703604:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576093.703607:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864395264:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576093.703634:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395392:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576093.703678:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576093.703693:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff92fc67049500 x1875449864395392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576093.703704:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049500 x1875449864395392/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576093.703711:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576093.703715:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576093.703798:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395392:192.168.202.132@tcp:38:kworker.0 00000100:00080000:0.0:1788576093.704670:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576093.704676:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576093.704679:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576093.704685:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7844 00000100:00080000:0.0:1788576093.704692:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576093.704698:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576093.704709:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576093.705286:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704bb80 x1875449864395520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576093.705352:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395520:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576093.705397:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:0.0:1788576093.705425:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576093.705430:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576093.705432:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576093.705437:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7844 00000100:00080000:0.0:1788576093.705441:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576093.705445:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576093.705452:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576093.705466:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049180 x1875449864395648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576093.706422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff92fc6704bb80 x1875449864395520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576093.706440:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704bb80 x1875449864395520/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576093.706449:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576093.706453:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576093.706456:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395520:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576093.706477:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395648:192.168.202.132@tcp:8:kworker.0 00000100:00080000:0.0:1788576093.706619:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576093.706624:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576093.706626:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576093.706631:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7844 00000100:00080000:0.0:1788576093.706637:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576093.706641:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576093.706647:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576093.706671:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048e00 x1875449864395776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788576093.706765:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576093.706780:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff92fc67049180 x1875449864395648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576093.706793:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049180 x1875449864395648/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576093.706800:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576093.706803:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576093.706817:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395648:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576093.706831:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395776:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576093.706852:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576093.706864:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff92fc67048e00 x1875449864395776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576093.706876:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048e00 x1875449864395776/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576093.706883:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576093.706886:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576093.706888:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864395776:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576094.149022:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576094.212849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576095.172918:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576095.237212:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576096.196963:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576096.264041:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576097.220919:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576097.284853:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576098.244987:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576098.308840:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576098.820842:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576098.820854:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576098.820858:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576098.820870:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7849 00000100:00080000:0.0:1788576098.820880:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576098.820887:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576098.820927:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049c00 x1875449864395904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576098.820959:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576098.820962:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576098.820964:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576098.820966:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7849 00000100:00080000:0.0:1788576098.820969:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576098.820972:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576098.820979:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576098.820989:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864396032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576098.820995:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576098.820998:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576098.820999:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576098.821001:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7849 00000100:00080000:0.0:1788576098.821003:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576098.821005:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576098.821007:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576098.821395:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864395904:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576098.821448:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576098.821490:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff92fc67049c00 x1875449864395904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576098.821507:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049c00 x1875449864395904/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576098.821518:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576098.821522:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576098.821526:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864395904:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576098.821532:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396032:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576098.821550:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576098.821560:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff92fc67049880 x1875449864396032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576098.821571:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864396032/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576098.821585:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576098.821588:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576098.821591:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396032:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576098.822115:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864396160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576098.822136:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576098.822139:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576098.822141:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576098.822146:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7849 00000100:00080000:0.0:1788576098.822150:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576098.822154:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576098.822160:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576098.822171:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864396288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576098.822179:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576098.822183:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576098.822184:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576098.822188:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7849 00000100:00080000:0.0:1788576098.822192:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576098.822195:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576098.822199:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576098.822208:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864396416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576098.822374:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396160:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576098.822395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576098.822406:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff92fc6704ad80 x1875449864396160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576098.822417:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864396160/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576098.822425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576098.822428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576098.822430:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396160:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576098.822434:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396288:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576098.822447:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576098.822454:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff92fc6704a680 x1875449864396288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576098.822461:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864396288/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576098.822465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576098.822467:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576098.822469:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396288:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576098.822472:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396416:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576098.822482:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576098.822487:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff92fc6704aa00 x1875449864396416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576098.822492:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864396416/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576098.822497:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576098.822498:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576098.822500:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396416:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576099.268882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576099.332817:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576100.293661:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576100.356861:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576101.316883:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576101.380835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576102.340950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576102.404981:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576103.365428:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576103.429035:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576103.940846:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576103.940859:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576103.940863:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576103.940874:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7854 00000100:00080000:0.0:1788576103.940884:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576103.940904:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576103.940939:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864396544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576103.940969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576103.940973:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576103.940975:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576103.940979:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7854 00000100:00080000:0.0:1788576103.940983:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576103.940987:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576103.940996:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576103.941008:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864396672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576103.941017:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576103.941020:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576103.941022:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576103.941026:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7854 00000100:00080000:0.0:1788576103.941029:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576103.941032:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576103.941037:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576103.941049:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864396800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576103.941058:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576103.941061:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576103.941062:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576103.941066:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7854 00000100:00080000:0.0:1788576103.941068:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576103.941071:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576103.941076:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576103.941086:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864396928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576103.941113:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576103.941117:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576103.941118:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576103.941122:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7854 00000100:00080000:0.0:1788576103.941126:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576103.941129:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576103.941133:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576103.941143:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864397056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576103.941488:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864396544:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576103.941547:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576103.941601:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff92fc6704a300 x1875449864396544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576103.941619:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864396544/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576103.941630:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576103.941633:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576103.941637:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864396544:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576103.941644:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396672:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576103.941667:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576103.941681:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff92fc67048a80 x1875449864396672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576103.941692:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864396672/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576103.941702:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576103.941705:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576103.941709:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396672:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576103.941716:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396800:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576103.941773:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576103.941784:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff92fc67048700 x1875449864396800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576103.941796:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864396800/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576103.941804:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576103.941807:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576103.941811:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396800:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576103.941817:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396928:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576103.941837:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576103.941848:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff92fc67048380 x1875449864396928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576103.941859:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864396928/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576103.941867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576103.941870:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576103.941873:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864396928:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576103.941878:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397056:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576103.941911:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576103.941922:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff92fc67048000 x1875449864397056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576103.941933:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864397056/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576103.941940:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576103.941943:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576103.941946:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397056:192.168.202.132@tcp:8:kworker.0 00010000:00010000:2.0:1788576104.392507:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576104.452857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576105.412925:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576105.476879:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576106.438598:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576106.500822:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576107.460886:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576107.524849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576108.488990:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576108.548845:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576109.061283:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576109.061288:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576109.061291:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576109.061299:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7859 00000100:00080000:0.0:1788576109.061307:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576109.061312:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576109.061337:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864397184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576109.061355:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576109.061370:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576109.061372:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576109.061376:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7859 00000100:00080000:0.0:1788576109.061379:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576109.061383:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576109.061392:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576109.061403:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864397312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576109.061413:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576109.061416:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576109.061417:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576109.061421:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7859 00000100:00080000:0.0:1788576109.061424:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576109.061428:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576109.061432:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576109.061439:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864397440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576109.061445:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576109.061447:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576109.061448:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576109.061449:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7859 00000100:00080000:0.0:1788576109.061451:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576109.061453:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576109.061456:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576109.061461:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864397568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576109.061466:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576109.061468:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576109.061469:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576109.061470:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7859 00000100:00080000:0.0:1788576109.061472:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576109.061473:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576109.061475:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576109.061480:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864397696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576109.061498:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864397184:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576109.061540:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576109.061568:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576109/real 1788576109] req@ffff92fc67049f80 x1875449864397184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576109.061584:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864397184/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576109.061593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576109.061596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576109.061599:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864397184:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576109.061605:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397312:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576109.061626:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576109.061637:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576109/real 1788576109] req@ffff92fc67048000 x1875449864397312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576109.061649:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864397312/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576109.061657:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576109.061660:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576109.061663:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397312:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576109.061668:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397440:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576109.061687:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576109.061697:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576109/real 1788576109] req@ffff92fc67048380 x1875449864397440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576109.061708:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864397440/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576109.061715:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576109.061716:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576109.061718:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397440:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576109.061739:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397568:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576109.061755:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576109.061762:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576109/real 1788576109] req@ffff92fc67048700 x1875449864397568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576109.061773:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864397568/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576109.061777:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576109.061779:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576109.061781:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397568:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576109.061784:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397696:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576109.061814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576109.061821:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576109/real 1788576109] req@ffff92fc67048a80 x1875449864397696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576109.061827:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864397696/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576109.061832:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576109.061834:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576109.061836:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397696:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576109.509029:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576109.572846:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576110.532891:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576110.596862:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576111.556877:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576111.620995:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:2.0:1788576112.581424:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576112.647883:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576113.604876:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576113.668878:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576114.180830:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576114.180838:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576114.180841:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576114.180850:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7864 00000100:00080000:0.0:1788576114.180859:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576114.180865:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576114.180903:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048a80 x1875449864397824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576114.180934:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576114.180937:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576114.180938:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576114.180941:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7864 00000100:00080000:0.0:1788576114.180944:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576114.180946:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576114.180954:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576114.180963:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048700 x1875449864397952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576114.180969:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576114.180972:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576114.180973:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576114.180977:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7864 00000100:00080000:0.0:1788576114.180980:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576114.180983:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576114.180987:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576114.180996:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048380 x1875449864398080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576114.181005:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576114.181007:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576114.181009:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576114.181011:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7864 00000100:00080000:0.0:1788576114.181014:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576114.181017:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576114.181022:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576114.181030:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67048000 x1875449864398208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576114.181037:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576114.181039:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576114.181041:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576114.181044:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7864 00000100:00080000:0.0:1788576114.181047:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576114.181050:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576114.181053:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576114.181062:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049f80 x1875449864398336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576114.181146:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864397824:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576114.181207:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576114.181261:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576114/real 1788576114] req@ffff92fc67048a80 x1875449864397824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576114.181279:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048a80 x1875449864397824/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576114.181288:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576114.181292:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576114.181296:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864397824:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576114.181303:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397952:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576114.181323:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576114.181335:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576114/real 1788576114] req@ffff92fc67048700 x1875449864397952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576114.181345:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048700 x1875449864397952/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576114.181353:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576114.181357:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576114.181359:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864397952:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576114.181364:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398080:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576114.181414:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576114.181425:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576114/real 1788576114] req@ffff92fc67048380 x1875449864398080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576114.181436:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048380 x1875449864398080/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576114.181443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576114.181446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576114.181449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398080:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576114.181454:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398208:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576114.181472:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576114.181483:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576114/real 1788576114] req@ffff92fc67048000 x1875449864398208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576114.181493:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67048000 x1875449864398208/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576114.181500:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576114.181503:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576114.181506:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67048000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398208:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576114.181512:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398336:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576114.181528:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576114.181538:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576114/real 1788576114] req@ffff92fc67049f80 x1875449864398336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576114.181548:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049f80 x1875449864398336/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576114.181555:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576114.181557:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576114.181560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398336:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576114.628900:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576114.692852:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576115.653068:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576115.717704:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576116.676920:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576116.741056:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576117.700897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576117.764871:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576118.726944:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576118.789574:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576119.300960:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576119.300973:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576119.300977:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576119.300985:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7869 00000100:00080000:1.0:1788576119.300993:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576119.300998:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576119.301031:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728aa00 x1875449864398464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576119.301057:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576119.301061:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576119.301063:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576119.301067:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7869 00000100:00080000:1.0:1788576119.301071:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576119.301074:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576119.301083:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576119.301095:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b480 x1875449864398592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576119.301103:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576119.301106:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576119.301107:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576119.301110:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7869 00000100:00080000:1.0:1788576119.301112:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576119.301115:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576119.301119:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576119.301127:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a300 x1875449864398720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576119.301135:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576119.301138:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576119.301139:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576119.301143:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7869 00000100:00080000:1.0:1788576119.301147:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576119.301150:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576119.301155:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576119.301163:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864398464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:1.0:1788576119.301170:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288e00 x1875449864398848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576119.301182:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576119.301186:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576119.301200:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576119.301205:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7869 00000100:00080000:1.0:1788576119.301209:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576119.301212:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576119.301218:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576119.301232:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288000 x1875449864398976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788576119.301235:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576119.301293:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576119/real 1788576119] req@ffff92fc6728aa00 x1875449864398464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576119.301311:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728aa00 x1875449864398464/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576119.301339:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576119.301344:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576119.301348:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864398464:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576119.301355:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398592:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576119.301378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576119.301422:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576119/real 1788576119] req@ffff92fc6728b480 x1875449864398592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576119.301435:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b480 x1875449864398592/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576119.301443:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576119.301446:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576119.301449:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398592:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576119.301455:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398720:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576119.301488:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576119.301499:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576119/real 1788576119] req@ffff92fc6728a300 x1875449864398720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576119.301510:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a300 x1875449864398720/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576119.301518:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576119.301522:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576119.301526:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398720:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576119.301551:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398848:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576119.301569:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576119.301578:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576119/real 1788576119] req@ffff92fc67288e00 x1875449864398848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576119.301588:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288e00 x1875449864398848/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576119.301595:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576119.301598:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576119.301601:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398848:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576119.301606:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398976:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576119.301621:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576119.301630:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576119/real 1788576119] req@ffff92fc67288000 x1875449864398976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576119.301639:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288000 x1875449864398976/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576119.301646:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576119.301648:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576119.301651:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864398976:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576119.748897:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576119.812860:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576120.774080:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576120.836856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576121.796832:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576121.860857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576122.820981:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576122.885529:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576123.846135:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576123.909551:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576124.421577:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576124.421589:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576124.421594:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576124.421606:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7875 00000100:00080000:2.0:1788576124.421651:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576124.421661:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576124.421749:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ca80 x1875449864399104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576124.421793:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576124.421798:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576124.421800:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576124.421804:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7875 00000100:00080000:2.0:1788576124.421808:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576124.421812:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576124.421826:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576124.421841:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f480 x1875449864399232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576124.421850:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576124.421853:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576124.421855:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576124.421858:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7875 00000100:00080000:2.0:1788576124.421862:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576124.421865:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576124.421876:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576124.421893:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c700 x1875449864399360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576124.421901:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576124.421904:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576124.421906:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576124.421910:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7875 00000100:00080000:2.0:1788576124.421913:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576124.421917:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576124.421925:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576124.421936:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ed80 x1875449864399488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576124.421944:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576124.421946:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576124.421948:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576124.421951:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7875 00000100:00080000:2.0:1788576124.421954:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576124.421957:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576124.421962:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576124.421971:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617dc00 x1875449864399616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576124.422144:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864399104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576124.422242:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576124.422272:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576124.422291:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576124.422312:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576124.872277:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576124.932824:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576125.893384:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576125.958082:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576126.917377:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576126.981394:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000400:00000100:1.0:1788576127.493115:0:2382:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.132:988 00000400:00000100:1.0:1788576127.493158:0:2382:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.132@tcp at host 192.168.202.132 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788576127.493172:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788576127.493305:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788576127.493317:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788576127.493325:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000800:00000100:1.0:1788576127.493332:0:2382:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.132@tcp 00000100:00100000:3.0:1788576127.493674:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576124/real 1788576127] req@ffff92fc4617ca80 x1875449864399104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576127.493700:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ca80 x1875449864399104/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576127.493711:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576127.493715:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576127.493757:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864399104:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576127.493766:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576124/real 1788576127] req@ffff92fc4617f480 x1875449864399232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576127.493778:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f480 x1875449864399232/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576127.493787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576127.493791:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576127.493794:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399232:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576127.493799:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576124/real 1788576127] req@ffff92fc4617c700 x1875449864399360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576127.493810:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c700 x1875449864399360/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576127.493818:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576127.493822:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576127.493825:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399360:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576127.493831:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576124/real 1788576127] req@ffff92fc4617ed80 x1875449864399488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576127.493842:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ed80 x1875449864399488/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576127.493850:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576127.493853:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576127.493856:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399488:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576127.493862:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576124/real 1788576127] req@ffff92fc4617dc00 x1875449864399616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576127.493873:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617dc00 x1875449864399616/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576127.493880:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576127.493883:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576127.493885:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399616:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576127.940882:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576128.005059:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576128.965489:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576129.028868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576129.540867:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576129.540879:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576129.540883:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576129.540894:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7880 00000100:00080000:2.0:1788576129.540905:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576129.540912:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576129.540946:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f100 x1875449864399744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576129.540978:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576129.540983:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576129.540985:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576129.540988:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7880 00000100:00080000:2.0:1788576129.540992:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576129.540995:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576129.541005:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576129.541016:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617f800 x1875449864399872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576129.541024:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576129.541027:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576129.541029:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576129.541032:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7880 00000100:00080000:2.0:1788576129.541035:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576129.541037:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576129.541041:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576129.541048:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864400000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576129.541055:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576129.541058:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576129.541060:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576129.541062:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7880 00000100:00080000:2.0:1788576129.541065:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576129.541067:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576129.541071:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576129.541078:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864400128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576129.541086:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576129.541089:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576129.541090:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576129.541093:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7880 00000100:00080000:2.0:1788576129.541095:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576129.541098:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576129.541102:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576129.541111:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864400256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576129.541264:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864399744:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576129.541327:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576129.541364:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576129/real 1788576129] req@ffff92fc4617f100 x1875449864399744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576129.541382:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f100 x1875449864399744/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576129.541393:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576129.541397:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576129.541401:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864399744:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576129.541408:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399872:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576129.541440:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576129.541451:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576129/real 1788576129] req@ffff92fc4617f800 x1875449864399872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576129.541462:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617f800 x1875449864399872/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576129.541469:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576129.541473:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576129.541476:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864399872:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576129.541481:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400000:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576129.541499:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576129.541509:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576129/real 1788576129] req@ffff92fc4617d180 x1875449864400000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576129.541520:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864400000/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576129.541534:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576129.541537:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576129.541540:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400000:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576129.541545:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400128:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576129.541565:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576129.541575:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576129/real 1788576129] req@ffff92fc4617c380 x1875449864400128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576129.541585:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864400128/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576129.541593:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576129.541596:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576129.541599:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400128:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576129.541604:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400256:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576129.541622:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576129.541633:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576129/real 1788576129] req@ffff92fc4617e680 x1875449864400256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576129.541644:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864400256/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576129.541651:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576129.541654:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576129.541658:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400256:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576129.988848:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576130.053093:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576131.014332:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576131.076863:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576132.036928:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576132.100851:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576133.062330:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576133.125648:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576134.084895:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576134.149098:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576134.661304:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576134.661314:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576134.661319:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576134.661329:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7885 00000100:00080000:1.0:1788576134.661339:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576134.661348:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576134.661384:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b100 x1875449864400384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576134.661417:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576134.661421:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576134.661422:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576134.661438:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7885 00000100:00080000:1.0:1788576134.661442:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576134.661446:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576134.661457:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576134.661468:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289880 x1875449864400512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576134.661478:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576134.661481:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576134.661484:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576134.661487:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7885 00000100:00080000:1.0:1788576134.661491:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576134.661504:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576134.661509:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576134.661519:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728ad80 x1875449864400640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576134.661528:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576134.661531:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576134.661533:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576134.661536:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7885 00000100:00080000:1.0:1788576134.661540:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576134.661543:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576134.661547:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576134.661556:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728b800 x1875449864400768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576134.661564:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576134.661567:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576134.661568:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576134.661571:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7885 00000100:00080000:1.0:1788576134.661574:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576134.661578:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576134.661582:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576134.661590:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289f80 x1875449864400896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576134.663309:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864400384:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576134.663378:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576134.663682:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff92fc6728b100 x1875449864400384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576134.663703:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b100 x1875449864400384/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576134.663714:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576134.663719:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576134.663760:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864400384:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576134.663768:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400512:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576134.663794:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576134.663807:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff92fc67289880 x1875449864400512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576134.663819:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289880 x1875449864400512/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576134.663828:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576134.663831:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576134.663834:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400512:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576134.663841:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400640:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576134.663861:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576134.663871:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff92fc6728ad80 x1875449864400640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576134.663881:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728ad80 x1875449864400640/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576134.663888:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576134.663891:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576134.663894:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400640:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576134.663899:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400768:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576134.663928:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576134.663938:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff92fc6728b800 x1875449864400768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576134.663948:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728b800 x1875449864400768/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576134.663956:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576134.663959:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576134.663962:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400768:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576134.663967:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400896:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576134.663984:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576134.663993:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff92fc67289f80 x1875449864400896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576134.664003:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289f80 x1875449864400896/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576134.664021:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576134.664025:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576134.664028:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864400896:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576135.111521:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576135.172868:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576136.132884:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576136.196864:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576137.157877:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576137.220842:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576138.180901:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576138.244850:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576139.205673:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576139.268915:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:0.0:1788576139.781269:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576139.781280:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576139.781284:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576139.781294:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7890 00000100:00080000:0.0:1788576139.781305:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576139.781313:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:0.0:1788576139.781352:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a300 x1875449864401024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576139.781383:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576139.781387:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576139.781389:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576139.781393:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7890 00000100:00080000:0.0:1788576139.781396:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576139.781400:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576139.781409:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576139.781421:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704aa00 x1875449864401152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576139.781431:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576139.781433:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576139.781435:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576139.781438:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7890 00000100:00080000:0.0:1788576139.781451:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576139.781455:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576139.781460:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576139.781471:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704a680 x1875449864401280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576139.781480:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576139.781483:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576139.781485:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576139.781489:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7890 00000100:00080000:0.0:1788576139.781492:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576139.781496:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576139.781500:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576139.781510:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6704ad80 x1875449864401408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576139.781518:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576139.781521:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576139.781523:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576139.781526:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7890 00000100:00080000:0.0:1788576139.781528:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576139.781532:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:0.0:1788576139.781536:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576139.781545:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67049880 x1875449864401536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576139.781640:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864401024:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576139.781699:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576139.781760:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576139] req@ffff92fc6704a300 x1875449864401024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576139.781777:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a300 x1875449864401024/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576139.781787:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576139.781790:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576139.781793:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864401024:192.168.202.132@tcp:250:kworker.0 00000100:00100000:3.0:1788576139.781800:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401152:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576139.781822:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576139.781834:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576139] req@ffff92fc6704aa00 x1875449864401152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576139.781846:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704aa00 x1875449864401152/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576139.781854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576139.781857:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576139.781859:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401152:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576139.781866:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401280:192.168.202.132@tcp:38:kworker.0 00000800:00000100:3.0:1788576139.781885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576139.781895:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576139] req@ffff92fc6704a680 x1875449864401280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576139.781907:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704a680 x1875449864401280/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576139.781913:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576139.781916:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576139.781919:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401280:192.168.202.132@tcp:38:kworker.0 00000100:00100000:3.0:1788576139.781924:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401408:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576139.781942:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576139.781952:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576139] req@ffff92fc6704ad80 x1875449864401408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576139.781962:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6704ad80 x1875449864401408/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576139.781969:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576139.781972:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576139.781975:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6704ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401408:192.168.202.132@tcp:8:kworker.0 00000100:00100000:3.0:1788576139.781981:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401536:192.168.202.132@tcp:8:kworker.0 00000800:00000100:3.0:1788576139.781997:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576139.782007:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576139] req@ffff92fc67049880 x1875449864401536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576139.782018:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67049880 x1875449864401536/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576139.782025:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576139.782027:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576139.782030:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67049880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401536:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576140.228891:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576140.294656:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576141.252974:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576141.318437:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576142.277426:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576142.340913:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576143.301313:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576143.364831:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576144.324903:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576144.388857:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576144.901271:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576144.901282:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576144.901286:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576144.901297:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7895 00000100:00080000:2.0:1788576144.901308:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576144.901315:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576144.901352:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864401664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576144.902288:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864401664:192.168.202.132@tcp:250:kworker.0 00000800:00000100:3.0:1788576144.902347:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:3.0:1788576144.902417:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff92fc4617fb80 x1875449864401664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.902432:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576144.902437:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1788576144.902438:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864401664/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.902439:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576144.902447:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7895 00000100:00080000:2.0:1788576144.902463:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576144.902467:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576144.902471:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00080000:3.0:1788576144.902476:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:3.0:1788576144.902480:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864401664:192.168.202.132@tcp:250:kworker.0 00000008:00100000:2.0:1788576144.902486:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576144.902505:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864401792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576144.902520:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576144.902523:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576144.902525:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576144.902530:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7895 00000100:00080000:2.0:1788576144.902534:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576144.902538:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576144.902543:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576144.902554:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864401920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576144.902563:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576144.902567:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576144.902569:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576144.902573:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7895 00000100:00080000:2.0:1788576144.902576:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576144.902579:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576144.902584:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576144.902594:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864402048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576144.902609:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576144.902613:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576144.902615:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576144.902619:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7895 00000100:00080000:2.0:1788576144.902623:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576144.902626:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576144.902631:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576144.902640:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864402176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576144.902940:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401792:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576144.903184:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576144.903200:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff92fc4617c000 x1875449864401792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576144.903214:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864401792/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576144.903224:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576144.903227:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576144.903230:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401792:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576144.903236:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401920:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576144.903257:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576144.903268:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff92fc4617d880 x1875449864401920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576144.903278:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864401920/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576144.903286:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576144.903289:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576144.903292:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864401920:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576144.903298:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402048:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576144.903325:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576144.903343:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff92fc4617ea00 x1875449864402048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576144.903353:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864402048/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576144.903360:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576144.903363:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576144.903366:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402048:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576144.903371:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402176:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576144.903389:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576144.903398:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff92fc4617e300 x1875449864402176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576144.903423:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864402176/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576144.903432:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576144.903436:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576144.903439:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402176:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576145.350818:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576145.412796:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576146.372856:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576146.436847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576147.397407:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576147.460849:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576148.421764:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576148.484874:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576149.445032:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576149.508893:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576150.020893:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576150.020906:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576150.020912:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576150.020925:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7900 00000100:00080000:2.0:1788576150.020938:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576150.020947:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576150.020994:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d500 x1875449864402304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576150.021206:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864402304:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576150.021289:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00080000:2.0:1788576150.021326:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576150.021330:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576150.021332:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576150.021336:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7900 00000100:00080000:2.0:1788576150.021340:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576150.021344:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576150.021357:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576150.021370:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ce00 x1875449864402432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576150.021435:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576150/real 1788576150] req@ffff92fc4617d500 x1875449864402304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576150.021454:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d500 x1875449864402304/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576150.021465:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576150.021485:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576150.021489:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576150.021491:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576150.021495:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7900 00000100:00080000:2.0:1788576150.021499:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576150.021503:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576150.021508:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576150.021522:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e300 x1875449864402560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576150.021531:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576150.021534:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576150.021536:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576150.021539:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7900 00000100:00080000:2.0:1788576150.021543:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576150.021546:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576150.021552:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576150.021561:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617ea00 x1875449864402688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576150.021569:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576150.021572:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576150.021574:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576150.021578:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7900 00000100:00080000:2.0:1788576150.021582:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576150.021585:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576150.021590:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576150.021600:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d880 x1875449864402816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576150.021660:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576150.021665:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864402304:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576150.021693:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402432:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576150.021756:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576150.021772:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576150/real 1788576150] req@ffff92fc4617ce00 x1875449864402432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576150.021783:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ce00 x1875449864402432/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576150.021792:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576150.021795:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576150.021799:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402432:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576150.021805:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402560:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576150.021824:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576150.021835:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576150/real 1788576150] req@ffff92fc4617e300 x1875449864402560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576150.021846:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e300 x1875449864402560/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576150.021854:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576150.021857:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576150.021860:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402560:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576150.021866:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402688:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576150.021884:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576150.021893:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576150/real 1788576150] req@ffff92fc4617ea00 x1875449864402688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576150.021904:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617ea00 x1875449864402688/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576150.021911:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576150.021914:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576150.021918:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402688:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576150.021923:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402816:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576150.021941:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576150.021952:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576150/real 1788576150] req@ffff92fc4617d880 x1875449864402816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576150.021962:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d880 x1875449864402816/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576150.021970:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576150.021973:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576150.021976:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864402816:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576150.468860:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576150.533048:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576151.492941:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576151.556838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576152.516979:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576152.580838:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576153.541313:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576153.604855:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576154.565051:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576154.628837:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:1.0:1788576155.140898:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576155.140907:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576155.140912:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576155.140921:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7905 00000100:00080000:1.0:1788576155.140931:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576155.140939:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:1.0:1788576155.140974:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728a680 x1875449864402944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576155.141007:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576155.141011:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576155.141013:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576155.141017:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7905 00000100:00080000:1.0:1788576155.141020:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576155.141023:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576155.141033:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576155.141044:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67289c00 x1875449864403072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576155.141054:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576155.141057:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576155.141058:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576155.141062:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7905 00000100:00080000:1.0:1788576155.141065:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576155.141068:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576155.141074:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576155.141083:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288380 x1875449864403200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576155.141092:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576155.141095:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576155.141097:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576155.141100:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7905 00000100:00080000:1.0:1788576155.141103:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576155.141106:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576155.141111:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576155.141119:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6728bb80 x1875449864403328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576155.141127:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576155.141130:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576155.141131:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576155.141134:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7905 00000100:00080000:1.0:1788576155.141137:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576155.141140:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:1.0:1788576155.141144:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576155.141152:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67288700 x1875449864403456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576155.141237:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864402944:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576155.141291:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576155.141330:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576155/real 1788576155] req@ffff92fc6728a680 x1875449864402944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576155.141348:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728a680 x1875449864402944/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576155.141358:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576155.141362:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576155.141365:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864402944:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576155.141375:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403072:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576155.141395:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576155.141407:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576155/real 1788576155] req@ffff92fc67289c00 x1875449864403072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576155.141418:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67289c00 x1875449864403072/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576155.141425:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576155.141428:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576155.141431:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67289c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403072:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576155.141437:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403200:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576155.141454:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576155.141464:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576155/real 1788576155] req@ffff92fc67288380 x1875449864403200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576155.141475:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288380 x1875449864403200/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576155.141496:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576155.141500:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576155.141502:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403200:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576155.141508:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403328:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576155.141527:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576155.141537:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576155/real 1788576155] req@ffff92fc6728bb80 x1875449864403328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576155.141547:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6728bb80 x1875449864403328/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576155.141554:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576155.141557:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576155.141560:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6728bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403328:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576155.141565:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403456:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576155.141582:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576155.141592:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576155/real 1788576155] req@ffff92fc67288700 x1875449864403456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576155.141603:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67288700 x1875449864403456/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576155.141610:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576155.141613:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576155.141616:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67288700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403456:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576155.588950:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576155.655614:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576156.612944:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576156.676956:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576157.637283:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576157.700847:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576158.660879:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576158.725057:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576159.685904:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576159.748936:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788576160.260944:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576160.260959:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576160.260965:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576160.260979:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7910 00000100:00080000:3.0:1788576160.260994:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576160.261004:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788576160.261057:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67339500 x1875449864403584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576160.261099:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576160.261103:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576160.261104:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576160.261109:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7910 00000100:00080000:3.0:1788576160.261112:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576160.261116:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576160.261128:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576160.261141:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a300 x1875449864403712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576160.261150:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576160.261153:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576160.261155:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576160.261158:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7910 00000100:00080000:3.0:1788576160.261162:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576160.261166:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576160.261170:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576160.261181:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338000 x1875449864403840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576160.261189:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576160.261192:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576160.261194:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576160.261197:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7910 00000100:00080000:3.0:1788576160.261200:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576160.261204:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576160.261210:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576160.261220:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864403968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576160.261228:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576160.261231:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576160.261232:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576160.261236:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7910 00000100:00080000:3.0:1788576160.261239:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576160.261242:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576160.261246:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576160.261257:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864404096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576160.261368:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864403584:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576160.261448:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576160.261508:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576160/real 1788576160] req@ffff92fc67339500 x1875449864403584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576160.261529:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67339500 x1875449864403584/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576160.261540:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576160.261544:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576160.261548:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67339500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864403584:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576160.261556:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403712:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576160.261580:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576160.261594:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576160/real 1788576160] req@ffff92fc6733a300 x1875449864403712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576160.261607:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a300 x1875449864403712/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576160.261616:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576160.261619:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576160.261622:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403712:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576160.261629:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403840:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576160.261649:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576160.261660:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576160/real 1788576160] req@ffff92fc67338000 x1875449864403840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576160.261672:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338000 x1875449864403840/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576160.261680:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576160.261684:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576160.261687:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403840:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576160.261694:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403968:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576160.261714:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576160.261759:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576160/real 1788576160] req@ffff92fc67338700 x1875449864403968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576160.261772:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864403968/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576160.261781:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576160.261784:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576160.261787:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864403968:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576160.261794:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404096:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576160.261814:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576160.261825:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576160/real 1788576160] req@ffff92fc6733bb80 x1875449864404096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576160.261836:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864404096/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576160.261843:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576160.261847:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576160.261850:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404096:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576160.709365:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576160.772909:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576161.734316:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576161.796881:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576162.757865:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576162.820869:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576163.782070:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576163.845713:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576164.805796:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576164.868946:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:3.0:1788576165.383439:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576165.383455:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576165.383460:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576165.383474:0:110631:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7915 00000100:00080000:3.0:1788576165.383488:0:110631:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576165.383498:0:110631:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:3.0:1788576165.383585:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338e00 x1875449864404224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576165.383623:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576165.383630:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576165.383632:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576165.383637:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7915 00000100:00080000:3.0:1788576165.383641:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576165.383645:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576165.383656:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576165.383669:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733a680 x1875449864404352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576165.383679:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576165.383683:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576165.383684:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576165.383688:0:110631:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7915 00000100:00080000:3.0:1788576165.383691:0:110631:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576165.383695:0:110631:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576165.383699:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576165.383709:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733b100 x1875449864404480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576165.383752:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576165.383757:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576165.383758:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576165.383762:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7915 00000100:00080000:3.0:1788576165.383766:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576165.383770:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576165.383776:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576165.383789:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc6733bb80 x1875449864404608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576165.383799:0:110631:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576165.383802:0:110631:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576165.383804:0:110631:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576165.383807:0:110631:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7915 00000100:00080000:3.0:1788576165.383810:0:110631:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576165.383813:0:110631:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:3.0:1788576165.383817:0:110631:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576165.383828:0:110631:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc67338700 x1875449864404736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576165.384569:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864404224:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576165.384652:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576165.384708:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576165/real 1788576165] req@ffff92fc67338e00 x1875449864404224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576165.384760:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338e00 x1875449864404224/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576165.384771:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576165.384776:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576165.384780:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864404224:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576165.384787:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404352:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576165.384820:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576165.384835:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576165/real 1788576165] req@ffff92fc6733a680 x1875449864404352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576165.384847:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733a680 x1875449864404352/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576165.384855:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576165.384859:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576165.384862:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404352:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576165.384868:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404480:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576165.384902:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576165.384912:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576165/real 1788576165] req@ffff92fc6733b100 x1875449864404480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576165.384931:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733b100 x1875449864404480/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576165.384939:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576165.384942:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576165.384945:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404480:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576165.384952:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404608:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576165.384983:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576165.384994:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576165/real 1788576165] req@ffff92fc6733bb80 x1875449864404608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576165.385016:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc6733bb80 x1875449864404608/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576165.385024:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576165.385028:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576165.385031:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc6733bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404608:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576165.385037:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404736:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576165.385056:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576165.385066:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576165/real 1788576165] req@ffff92fc67338700 x1875449864404736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576165.385077:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc67338700 x1875449864404736/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576165.385090:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576165.385093:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576165.385096:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc67338700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404736:192.168.202.132@tcp:8:kworker.0 00010000:00010000:1.0:1788576165.829109:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576165.892956:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576166.852902:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576166.916835:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576167.876963:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576167.940954:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576168.901156:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576168.965043:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576169.925372:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576169.989324:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00000100:00080000:2.0:1788576170.500899:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) b62659d6-7f40-4480-a04f-ba5899daf967->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576170.500910:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576170.500914:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576170.500925:0:31488:0:(import.c:545:import_select_connection()) MGC192.168.202.132@tcp: connect to NID 192.168.202.132@tcp last attempt 7921 00000100:00080000:2.0:1788576170.500936:0:31488:0:(import.c:630:import_select_connection()) MGC192.168.202.132@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576170.500943:0:31488:0:(import.c:665:import_select_connection()) MGC192.168.202.132@tcp: import ffff92fc4536c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000100:00100000:2.0:1788576170.500986:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c000 x1875449864404864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576170.501019:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576170.501023:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576170.501025:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576170.501037:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7921 00000100:00080000:2.0:1788576170.501041:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576170.501045:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff92fc67229000: import ffff92fc48dc5800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576170.501057:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576170.501070:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617fb80 x1875449864404992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576170.501084:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576170.501087:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576170.501089:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576170.501092:0:31488:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7921 00000100:00080000:2.0:1788576170.501096:0:31488:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576170.501100:0:31488:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff92fc67229000: import ffff92fc5824c800 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576170.501104:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576170.501114:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617e680 x1875449864405120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576170.501122:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576170.501125:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576170.501127:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576170.501131:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7921 00000100:00080000:2.0:1788576170.501134:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576170.501137:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff92fc67229000: import ffff92fc5824d000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576170.501141:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576170.501150:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617c380 x1875449864405248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576170.501170:0:31488:0:(pinger.c:229:ptlrpc_pinger_process_import()) a74e4397-7fb3-41e0-aa97-867f5c09bb42->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576170.501173:0:31488:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576170.501175:0:31488:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576170.501178:0:31488:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: connect to NID 192.168.202.132@tcp last attempt 7921 00000100:00080000:2.0:1788576170.501182:0:31488:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576170.501185:0:31488:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff92fc67229000: import ffff92fc58248000 using connection 192.168.202.132@tcp/192.168.202.132@tcp 00000008:00100000:2.0:1788576170.501190:0:31488:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576170.501211:0:31488:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff92fc4617d180 x1875449864405376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576170.501581:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864404864:192.168.202.132@tcp:250:kworker.0 00000800:00000100:0.0:1788576170.501648:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576170.501692:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576170/real 1788576170] req@ffff92fc4617c000 x1875449864404864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576170.501709:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c000 x1875449864404864/t0(0) o250->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576170.501746:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc4536c800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576170.501751:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576170.501755:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b62659d6-7f40-4480-a04f-ba5899daf967:2392:1875449864404864:192.168.202.132@tcp:250:kworker.0 00000100:00100000:0.0:1788576170.501762:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404992:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576170.501784:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576170.501797:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576170/real 1788576170] req@ffff92fc4617fb80 x1875449864404992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576170.501808:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617fb80 x1875449864404992/t0(0) o38->lustre-MDT0000-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576170.501816:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc48dc5800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576170.501819:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576170.501822:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864404992:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576170.501828:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864405120:192.168.202.132@tcp:38:kworker.0 00000800:00000100:0.0:1788576170.501846:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576170.501855:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576170/real 1788576170] req@ffff92fc4617e680 x1875449864405120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576170.501862:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617e680 x1875449864405120/t0(0) o38->lustre-MDT0001-mdc-ffff92fc67229000@192.168.202.132@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576170.501867:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824c800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576170.501869:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576170.501871:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864405120:192.168.202.132@tcp:38:kworker.0 00000100:00100000:0.0:1788576170.501874:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864405248:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576170.501885:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576170.501890:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576170/real 1788576170] req@ffff92fc4617c380 x1875449864405248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576170.501896:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617c380 x1875449864405248/t0(0) o8->lustre-OST0000-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576170.501900:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc5824d000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576170.501902:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576170.501903:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864405248:192.168.202.132@tcp:8:kworker.0 00000100:00100000:0.0:1788576170.501906:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864405376:192.168.202.132@tcp:8:kworker.0 00000800:00000100:0.0:1788576170.501915:0:2392:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.132@tcp 00000100:00100000:0.0:1788576170.501920:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576170/real 1788576170] req@ffff92fc4617d180 x1875449864405376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576170.501926:0:2392:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff92fc4617d180 x1875449864405376/t0(0) o8->lustre-OST0001-osc-ffff92fc67229000@192.168.202.132@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576170.501930:0:2392:0:(import.c:65:import_set_state_nolock()) ffff92fc58248000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576170.501932:0:2392:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.132@tcp failed (-110) 00000100:00100000:0.0:1788576170.501933:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff92fc4617d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:a74e4397-7fb3-41e0-aa97-867f5c09bb42:2392:1875449864405376:192.168.202.132@tcp:8:kworker.0 00010000:00010000:0.0:1788576170.948867:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576171.012858:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:1.0:1788576171.972989:0:112078:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576172.036856:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds 00010000:00010000:0.0:1788576172.996905:0:112077:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576173.060899:0:112540:0:(client.c:2716:ptlrpc_set_wait()) set ffff92fc49a8ab40 going to sleep for 0 seconds Debug log: 78548 lines, 78548 kept, 0 dropped, 0 bad.